# How to use parfor with a range from a very large object and cause memory problems

2 views (last 30 days)
Jon Ward on 18 Nov 2021
Edited: Jon Ward on 18 Nov 2021
How can I achieve the following without gettting the broadcast variable warning and minimising memory usage by Matlab trying to copy the full variable R?
This is a simplified example of what I am trying to achieve, but the memory requirements explode when I try to do this (I am using 48 processors in parallel on a high spec AWS box with lots of memory).
Thanks!
T = 10000;
S = 100; %S is actually ~1000
P = 10;
M = 60;
R = rand(T,S,P,M); % This is an example, R and C below get populated from another process, but this is to illustrate the size of R.
C = rand(T,M);
resultsM = zeros(T,S,P);
resultsSD = zeros(T,S,P);
parfor t=50:T
r = R(1:t,:,:,:); % Broadcast warning here
% Do some stuff with r
c = C(1:t,:);
for s=1:S
for p=1:P
rr = squeeze(r(:,s,p,:));
tmpResults = sum(c.*rr,2);
resultsM(t,s,p) = mean(tmpResults); % Mean and below sd are examples, I apply other custom functions to tmpResults
resultsSD(t,s,p) = std(tmpResults);
end
end
end
##### 2 CommentsShowHide 1 older comment
Jon Ward on 18 Nov 2021
Thanks, i have just edited the code to hopefully show more of the type of thing I am trying to do.

Raymond Norris on 18 Nov 2021
Have you considered assigning R in the parfor?
parfor t = 1:T
R = rand(T,S,P,M)
...
end
It's less intuitive for MATLAB serial code, but may be a better practice for MATLAB parallel code. R is now a temporary variable, which can't be referenced after the parfor.
Also look at ticBytes and tocBytes to see how much data is truely getting passed back and forth.
Jon Ward on 18 Nov 2021
Unfortunately not, I was using rand to show the size of R only, it actually gets generated from another process. I have edited the code above to show what I am trying to do.

R2020b

### Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!