How to create data file and copying data on it for parallel programing in MATLAB?

I want to run an ODE system by varying a suitable parameter, suppose \alpha. Now I have divided the job into the corresponding worker and created a data file so that for each parameter \alpha, I need to copy a certain measure of the system into the data file. When I googled it and went through some documentation, I realized that using a single file only creates a route to a corrupt file. It is not a way to do that, and I need to create multiple files. I need to ask whether there is any way to create files with some exact name (must be in parallel programing) and scan data from all the files to create another data file that contains all the data.

Answer 1

[Edric Ellis]

You are quite right that trying to write to the same file from multiple workers will cause problems. Here's one way that you could get the workers to write to unique files, and then get the client to amalgamate those files:

alphaValues = 0:.01:1;
numValues = numel(alphaValues);
parfor ii = 1:numValues
    % generate a unique file name to write to
    fname = sprintf('data_%d', ii);
    % Run a simulation for this value of alpha
    data = runMySimulation(alphaValues(ii));
    % save the data - note workaround for SAVE inside PARFOR
    iSaveData(fname, data)
end

% At this point, files data_1 .. data_(numValues) exist. We can read them
% at the client
for ii = 1:numValues
    fname = sprintf('data_%d', ii);
    s = load(fname);
    allData{ii} = s.data;
end
% Save all of the data in one big data file
save('oneBigDataFile', 'allData')

% Workaround for SAVE inside PARFOR
function iSaveData(fname, data)
save(fname, 'data')
end

[Deeptajyoti Sen]

Thank you very much for your suggestion. I am going through that. First time when I ran the code, it is perfectly ok. But when I ran it second time it shows some error that "Unable to write file data_1: No such file or directory." in isave data.

In a particular worker, when some data file is created, I have to open it several times to print the data, as inside the parfor loop there is more for loops are there with conditional statement. So, the old data are lost. So, I have used "save(filename,variables,'-append','-nocompression')" this format inside the function 'iSaveData'.

Actually, here my "data" is two colums vector, one is alphavalues and another is some measure. So I chane function 'iSaveData' in to "save(fname, 'data1','data2')". But when, the first time I have open the file, I saw that data are same one after another, i.e. at first data1, below that data2 and previous data are lost. I need something like "data1 data2".

[Edric Ellis]

Hm, without some more detailed reproduction steps, it's a bit hard to know exactly what's going wrong.

Unless the data that you're saving is really large (many GB in total), then you might be better off simply returning the data from the parfor loop, and letting the client save everything. I.e.

parfor ii = 1:numValues
    data1{ii} = someFcn(ii);
    data2{ii} = someOtherFcn(ii);
end
save allData data1 data2 % or whatever.

[Deeptajyoti Sen]

I have inserted the explicit code.

[Edric Ellis]

There's a lot going on there, and I haven't worked through it all in detail, but I'd definitely recommend at the very least setting up the arrays that you want to save ahead of time, rather than calling iSaveData multiple times in inner nested for loops. You should be able to end up with a single call to iSaveData per parfor loop iteration. Your code will probably be quite a bit faster when it doesn't have to call save so many times.