MATLAB Answers

Germán
0

parfor - Attempt to serialize data which is too large

Asked by Germán
on 2 Apr 2012
Latest activity Commented on by Edric Ellis
on 8 Nov 2013
I'm working on a large data set that is is stored as a cell array X, where each cell contains an array of about 2MB, totaling 1GB. I'm running it in parallel on 50 cores that are each on separate nodes. I get the error "Attempt to serialize data which is too large" when I try to run the whole array . When I run a small array X I don't get the error. I check on the nodes and there's plenty of memory left, it seems the bottleneck is data transmission
I'm using the following code
Y1= cell(1,N1);
Y2 = cell(1,N1);
parfor i = 1:N1
XX = X{i};
Np = size(XX,1);
Y1_temp = zeros(400, Np-1);
Y2_temp = zeros(400, Np-1);
for j= 1:(Np-1)
Y1_temp(:,j) = someFunction;
Y2_temp(:,j) = someOtherFunction;
end
Y1{i} = Y1_temp;
Y2{i} = Y2_temp;
end
I don't understand why this is, has anyone ever got this error? I couln't find anything online. I also tried writing it as an spmd block where I operate on the local worker data but I get the same error. Any help will be very appreciated.
Regards Germán

  0 Comments

Sign in to comment.

3 Answers

Answer by Edric Ellis
on 8 Nov 2013
 Accepted Answer

The 2GB PARFOR transmission limitation was removed from Parallel Computing Toolbox in R2013a.

  0 Comments

Sign in to comment.


Answer by Konrad Malkowski on 2 Apr 2012

There is a 2GB transmission limit between client and workers. Could you provide me with additional values for Np, N1, Nt, etc so that I have a better picture of which variables are causing the problem?

  1 Comment

sure, the values are usually Np=100000, N1=Nt=512.
My understanding is that only the sliced portions of X, Y1, Y2 would be transmitted in each iteration, is that correct? If so the the size of a single cell in those variables is never above 2 MB.
Thanks for your help!

Sign in to comment.


Answer by James
on 4 Nov 2013

I have the same problem. I think it is a common problem, but I couldn't find a solution for this. If it is the 2GB transmission limit, then how can we get around this then?

  2 Comments

I got around it using normal arrays instead of cell arrays since there was a bug with cell arrays, but the issue might have been solved in the latest releases.
The 2GB PARFOR transmission limitation was removed from Parallel Computing Toolbox in R2013a.

Sign in to comment.