MATLAB Answers

Saving values in parfor loops

51 views (last 30 days)
Ronaldo
Ronaldo on 17 Apr 2013
I have a large database (1e8,4) and I want to use parfor command to do some calculations. I also want to use matlabpool command. My problem is in saving the output of the calculations. I used the following line to save the outputs:
parsave(sprintf('%d.mat', i), NumCr, NumNi);
where parsave is a function to save variables NumCr and NumNi.
The problem is that since this line of the code creates a huge number of *.mat files, MATLAB crashes. I also know that with matlabpool open command, global variables are not working. Is it possible to mention how I can save the values?

  2 Comments

Edric Ellis
Edric Ellis on 17 Apr 2013
Why do you want to save data from the workers - is it not sufficient to let the PARFOR loop complete and save the data from the client?
Are you creating one file per 1e8 iterations of your PARFOR loop?
Ronaldo
Ronaldo on 17 Apr 2013
I would appreciate it if you mention how I can modify the following code to save the values of NumCr and NumNi in each iteration?
parfor i=1:size(A,1)
O=A(i,1:4);
Om=ones(size(A,1),3);
Om1=O(1,1)*ones(size(A,1),1);
Om2=O(1,2)*ones(size(A,1),1);
Om3=O(1,3)*ones(size(A,1),1);
d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
distance=d>=R1m & d<=R2m;
NumCr=sum(distance.*Cr);
NumNi=sum(distance.*Ni);
parsave(sprintf('%d.mat', i), NumCr, NumNi);
end

Sign in to comment.

Accepted Answer

Thomas
Thomas on 17 Apr 2013
I assume it doesn't really crash with an error but it takes close to forever to output that many files...(?)
I don't know about the memory limitations you have .. your database should be around 1.5GB ... but can't you save the results in an matrix or a cell array first? Like
results = zeros(length(database)) %need to define this outside parfor
parfor i=1:length(database)
result(i) = NumNi;
end;
save...
Maybe you could even do this inplace, so database(i,1) = NumNi to save memory. Alternatively, if you run into memory problems you could divide your loop into smaller chunks and process some 10000 or 100000 elements at a time in parallel, then save and then continue.
If you have a lot of memory you could also try to setup a ramdisk and output your mat files on this disk - which is much faster even compared to an ssd disk.

  3 Comments

Ronaldo
Ronaldo on 17 Apr 2013
result(i)=NumNi is not working in parfor loops. Below is the code that I wrote. I would appreciate it if you help me understand what my mistake is.
parfor i=1:size(A,1)
O=A(i,1:4);
Om=ones(size(A,1),3);
Om1=O(1,1)*ones(size(A,1),1);
Om2=O(1,2)*ones(size(A,1),1);
Om3=O(1,3)*ones(size(A,1),1);
d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
distance=d>=R1m & d<=R2m;
NumCr=sum(distance.*Cr);
NumNi=sum(distance.*Ni);
parsave(sprintf('%d.mat', i), NumCr, NumNi);
end
Thomas
Thomas on 17 Apr 2013
There is no reason why result(i)=NumNi should not work with parfor. I just checked it does. However, it is crucial to define the whole result matrix/vector BEFORE the loop as i did above. Otherwise it would grow dynamically, which is not possible with parallel constructs.
Also try if your loop works in general with a small example database.
Ronaldo
Ronaldo on 17 Apr 2013
It works. Thanks a lot for your great help.

Sign in to comment.

More Answers (0)

Tags