Parfor loop: how to keep the temporary variables ?

Hello,

I am working with a time-consuming code and I want to put in parallel computing the independant functions I use.

I have the following script (only a fraction of it):

parfor i = 1:2
    
    if i == 1
        [T20cm_land, albedo_land, smb_land, swe_land, T1m_land, T0_land, abl_land,...
    acc_land, runoff_land] = display_output_land(point_output_path_land, yr, mm, dd, date);
    else
        [T20cm_glacier, albedo_glacier, smb_glacier, swe_glacier, T1m_glacier, T0_glacier,...
    abl_glacier, acc_glacier, runoff_glacier] = display_output_glacier(point_output_path_glacier, yr, mm, dd, date);
    end
end

These lines are themselves part of a function. In it, I use the resulting variables (eg: albedo_land, albedo_glacier etc..) to make some calculations and plots. Originally I just ran the functions by writing :

        [T20cm_land, albedo_land, smb_land, swe_land, T1m_land, T0_land, abl_land,...
    acc_land, runoff_land] = display_output_land(point_output_path_land, yr, mm, dd, date);

        [T20cm_glacier, albedo_glacier, smb_glacier, swe_glacier, T1m_glacier, T0_glacier,...
    abl_glacier, acc_glacier, runoff_glacier] = display_output_glacier(point_output_path_glacier, yr, mm, dd, date);

But these two functions are independant and take around 30 minutes to run. Is there a way to save all the output variables from these 2 functions outside of the parfor loop, in order to use them later in the script ?

Have a nice day !

Answer 1

[Edric Ellis]

Output variables from parfor must be either sliced or reduction variables. So, you could adapt your code like so:

parfor i = 1:2
    if i == 1
        [T20cm{i}, albedo{i}, . . .] = display_output_land(. . .);
    else
        [T20cm{i}, albedo{i}, . . .] = display_output_glacier(. . .);
    end
end

This converts all the outputs into sliced form.

An alternative given the fact that you really essentially have two completely separate function calls is to use parfeval instead, like this:

% Invoke both function evaluations in parallel.
fut_land = parfeval(@display_output_land, 9, point_output_path_land, yr, mm, dd, date);
fut_glacier = parfeval(@display_output_glacier, 9, point_output_path_glacier, yr, mm, dd, date);
% Collect results - this blocks until the evaluation is complete.
[T20cm_land, albedo_land, . . .] = fetchOutputs(fut_land);
[T20cm_glacier, albedo_glacier, . . .] = fetchOutputs(fut_glacier);
   

[V.D-C]

Thank you very much for your answer ! I actually found another way to deal with it. I saved my variables inside the functions themselves into temporary folders. Then, later in the script I load these variables and once everything is calculated, I delete the temporary file. It worked so far but I guess might not be as efficient as your solution.

[Edric Ellis]

Glad you got things working!

Saving files within parfor is generally not a great idea - and not just from an efficiency standpoint - it might work in a simple case like this providing you can ensure the workers are always writing to different files, but it's easy to trip up if you end up trying to write to the same files from different workers. Also, if you scaled up to a remote cluster, then there's a chance that the client MATLAB might not be able to see the same filesystem as the workers.

[V.D-C]

Then I should change my code for your solution, I'm working on a cluster (even though the files are written in a different folder). Thank you for your advices !