Can't find file on worker - Parallel Programming
9 views (last 30 days)
Show older comments
Hi, I'm attempting to parallelize part of my code to speed up the process. But have become stuck when one of the attached files cannot be found by a worker. The file is being opened and modified on each driver for calculation before being reverted back to it's original state. The missing file is the "hydroNet.NetworkFile". Do I have to create a duplicate file for each individual worker?
parpool('local','AttachedFiles',{'epanet2.dll',hydroNet.NetworkFile});
spmd
folder = getAttachedFilesFolder(hydroNet.NetworkFile);
if not(libisloaded('epanet2'))
loadlibrary('epanet2');
end
oldFolder = cd(folder); % Change to that folder
[OK,output] = system(x(hydroNet,pumpCombo,nCluster,pumpCluster,idxSysLink,logger));
cd(oldFolder); % Change back to the original folder
end
0 Comments
Answers (1)
Raymond Norris
on 11 Nov 2021
You mean to say that folder is empty?
At the top of the spmd, add this
spmd
getAttachedFilesFolder
hydroNet.NetworkFile
getAttachedFilesFolder(hydroNet.NetworkFile)
...
end
4 Comments
Raymond Norris
on 13 Nov 2021
You can have many processes reading the same file, but need to have semiphore or another way to have simultaneous writes (e.g. parallel database). I'm not sure why with one worker it will on occasion throw an error.
In regards to your situtation, each worker ought to be able to read from the same file, but then they might write to their own file, followed by a way to "true" up the output. Or maybe send everyone their local part and then write to one file. For example (I'm leaving off error handling)
spmd
% Import data
fid1 = fopen('input.txt','rt');
A = fscanf(fid1,'%s');
% Modify data
len = length(A);
variantB = A(randperm(len));
% Send each local part to everyone
replicatedB = gcat(variantB);
% Write to disk one output file
fid2 = fopen('output.txt','wt');
fprintf(fid2,[repmat('%c',1,len) '\n'],replicatedB);
fclose('all');
end
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!