MATLAB Answers

How do I use SAVE with a PARFOR loop using Parallel Computing Toolbox?

502 views (last 30 days)
I would like to save some variables to MAT files from inside a PARFOR loop. However I get an error:
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Transparency violation error.
See Parallel Computing Toolbox documentation about Transparency.

Accepted Answer

MathWorks Support Team
MathWorks Support Team on 24 Feb 2020
Edited: MathWorks Support Team on 24 Feb 2020
Transparency is violated by the SAVE command because in general MATLAB cannot determine which variables from the workspace will be saved to a file.
The solution is to move the call to SAVE to a separate function and to call that function from inside the PARFOR loop. Pass any variables that are to be saved as arguments to the function. That way MATLAB can determine which ones will be saved and transparency is not violated.
For example:
Save the following as "parsave.m":
function parsave(fname, x,y)
save(fname, 'x', 'y')
end
Then run it with:
parfor ii = 1:4
x = rand(10,10);
y = ones(1,3);
parsave(sprintf('output%d.mat', ii), x, y);
end
Similarly, for loading several files with enumerated file names, we can use the following code.
function [output] = par_load(i)
eval(['load file_' num2str(i) '.mat']);
output = time;
end
Which can be run with:
parfor i=1:4
data = par_load(i);
a(i) = 10 + data(1);
end

  4 Comments

Show 1 older comment
Dave Lee
Dave Lee on 22 Dec 2018
This parsave method works in 2018a. How about loading different files under different parfor loop iterations, any similar method?

Sign in to comment.

More Answers (6)

Daniel
Daniel on 14 Mar 2016
A nice alternative to implement your own save function is using the built-in matfile function:
parfor ii = 1:4
m=matfile(sprintf('output%d.mat', ii),'writable',true)
x = rand(10,10);
y = ones(1,3);
m.x=x;
m.y=y;
end
Here the parser has no issue understanding which variables are used.

Tihomir Kostadinov
Tihomir Kostadinov on 22 Aug 2014
Edited: Tihomir Kostadinov on 22 Aug 2014
I have enhanced the function that allows saving within a parfor loop as follows. This should allow giving it a variable number of arguments and the function recognizes their names. Let me know if there are problems with this implementation.
function save_for_parfor(fname,numvars,varargin)
for i = 1:numvars
eval([inputname(i+2),' = varargin{i};']);
end
save('-mat',fname,inputname(3));
for i = 2:numvars
save('-mat',fname,inputname(i+2),'-append');
end
This function would be called as follows:
>> x = 3; y = 4; z = 5; g = 'tester_parfor';
>> save_for_parfor('tester.mat',4,x,y,z,g)

  1 Comment

Walter Roberson
Walter Roberson on 13 Mar 2016
Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

Sign in to comment.


Qifeng Chen
Qifeng Chen on 30 Nov 2014
This is a better implementation based on Tihomir. The code here does not require the input of "numvars".
function parsave(fname,varargin)
numvars=numel(varargin);
for i=1:numvars
eval([inputname(i+1),'=varargin{i};']);
end
save('-mat',fname,inputname(2));
for i = 2:numvars
save('-mat',fname,inputname(i+1),'-append');
end
end

  5 Comments

Show 2 older comments
Linus Schumacher
Linus Schumacher on 23 May 2017
Erik's solution still give me a transparency violation error in Line 6 (savevar...)
Myrtle42
Myrtle42 on 16 Jun 2017
This still gives an error in Matlab R2016a: the variable I'm trying to save cannot be classified. Same with Alex Shaw's version below. Any ideas? Here is my code:
function test
var_to_save = NaN(10,1);
parfor ii = 1:10
var_to_save(ii,:) = rand;
parsave('test.mat',var_to_save)
end
end
function parsave(varargin)
savefile = varargin{1}; % first input argument
for i = 2:nargin
savevar.(inputname(i)) = varargin{i}; % other input arguments
end
save(savefile,'-struct','savevar')
end

Sign in to comment.


Alex Shaw
Alex Shaw on 9 Dec 2015
Edited: Alex Shaw on 9 Dec 2015
I encountered this issue, and used the function proposed by Qifeng Chen.
This worked very well in 2015a.
However, since upgrading to 2015b my code reports a transparancy violation again. The release notes state that transparancy is now enforced more strictly, however all variables that I attempt to save either appear as inputs in expression in the parfor loop, or are the results of calculation in the parfor loop so I think this may be a bug.
Further to this, the affected function seems to be inputname(). The simplest (and probably fastest) way to work round this is to add all variables to a single structure variable, and then save that with a fixed name:
function parsave_simple(fname,C )
save('-mat',fname,'C');
end
however, this would have been a lot of surgery in my case, plus other code expects the variables not to be in structure format. So the 'minimal surgery' option in my case is simply to pass the names of variables into the function via a cell array:
function parsave_named(fname____,vnames___,varargin)
numvars___=numel(varargin);
for ii___=1:numvars___
eval([vnames___{ii___},'=varargin{ii___};']);
end
save('-mat',fname____,vnames___{1});
for ii___ = 2:numvars___
save('-mat',fname____,vnames___{ii___},'-append');
end
end
This method seems a bit messy, there is a performance overhead to calling 'eval' and some very hard to trace bugs will arise if names do not match the actual variables. Also note the use of strange var names to ensure eval does not overwrite any internal logic! But on the other hand, it only needs one extra line in the calling loop to populate the names variable.

  2 Comments

echo zhang
echo zhang on 11 Apr 2019
This works for me, and a working example below:
parfor ii = 1:n
x=1;
y=2;
z=3;
var_name = ["x","y","z"];
parsave('whatever.mat',var_name,x,y,z);
end

Sign in to comment.


Anand
Anand on 13 Mar 2016
Edited: Anand on 13 Mar 2016
Another addition to various versions of doing this. This one does not use a structure. It, IMO, combines the best features of all answers above.
function parfor_save(varargin)
fname=varargin{1};
for i=2:nargin
eval([inputname(i),'=varargin{i};']);
if i==2
save('-mat',fname,inputname(i));
else
save('-mat',fname,inputname(i),'-append');
end
end

  1 Comment

Walter Roberson
Walter Roberson on 13 Mar 2016
Don't use eval(). Store the variables in a struct and use save with the -struct option. No -append is needed.

Sign in to comment.


CrossEntropy
CrossEntropy on 3 Dec 2019
The solution that works for me, on R2018a, is to define a function which includes the entire for loop, and then use save inside that function. Anything else I tried among the solutions in here id not work for me.

  1 Comment

Sign in to comment.

Sign in to answer this question.

Tags

Products


Release

R2010a