cplex object params not transferred to parfor loop

I am trying to solve the batch of CPLEX problems in parallel using parfor loop. I'm not very fluent in Matlab data types, but CPLEX problems seem to be stored as handle class objects with two structs: Model and Param.

Now, say I have an array with the paths to my .lp problems and I am reading them into cell array:

subp_array = {}; % array to store problems
nSubp = 1;       % for indexing problems

% read models into subp_array

for file = filepaths
    prob = Cplex('uc');
    prob.readModel(file{1});
    prob.Param.threads.Cur = 1;
    subp_array{nSubp} = prob;
    nSubp = nSubp+1;
end

The param threads.Cur is particularly important, because it indicates how many threads can be utilized for a single CPLEX problem (and with anything other than 1, it does not bring any time benefit). Now when I run the loop to solve the problems and I print the param value:

parfor subp_index = 1:length(subp_array)

    prob = subp_array{subp_index};
    disp(prob.Param.threads.Cur);
    
    % solve the problem
    prob.solve();

end

They all happen to be 0 (default value). If I run the for loop instead, it keeps the value of 1.

What happens here? I'm actually trying to catch more differences in my original code as parfor returns completely different results than for - but this one could be an indication for me where the differences come from.

Thanks

Answer 1

[Edric Ellis]

The most likely cause of the problem here is that the output of Cplex cannot be saved to a file and then loaded again. Transfer of data to the workers in a parallel pool is equivalent to calling save and then load (but the data isn't stored on disk at any point), and some data types in MATLAB do not correctly support save and load.

If you determine that the Cplex data types cannot be correctly saved and loaded, then the simplest workaround is to build the Cplex data directly on the worker, operate on it there, and return to the client only results (which are hopefully of a simple data type that can be correctly transferred). So, you might do something like this:

parfor idx = 1:numel(filepaths)
    prob = Cplex('uc');
    prob.readModel(filepaths{idx});
    prob.Param.threads.Cur = 1;
    prob.solve();
    out{idx} = doSomething(prob); % Not sure what you need here. 
end

This code assumes that the workers can see the same files as the client. This will be the case if you are using the 'local' cluster type (i.e. the default). If the workers can't see the same files, then you might need to use addAttachedFiles to make the files available on the workers.

[Katarzyna Furmanska]

OK, thank you, it explains a lot!

In fact, the CPLEX problems are one of the properties of the handle class objects, and there is quite a few of other properties that depend on the solution of CPLEX problem. In other words, I have a handle class objects A indexed over 1...n, where A(i) has, say, the following properties:

A(i).cpx, which is the CPLEX model

A(i).x, which gets the A(i).cpx.Solution.x values

and several other that depend on A(i).cpx, for i in 1...n.

A(i).x values calculated in for and parfor loops are different and I can imagine it somehow comes from what you have explained (not 100% sure what exactly is happenning though, because parfor is giving me different, but still nonzero results). I'm just wondering whether cell array works here for keeping the results, because there is quite a lot of them - is there any other handy way to do this?

Thanks a lot

[Edric Ellis]

One thing that you need to bear in mind is that when you use a handle object inside a parfor loop, each worker gets a copy of that object. Updates that you make on the worker are not automatically reflected back at the client. So perhaps that's what's going on. I'd still be looking to try and build the handle objects directly on the workers inside the parfor loop, and sending back to the client only the results - presuming they can safely be copied. More here in the doc.

[Katarzyna Furmanska]

Yeah, I've read the doc and I've already fixed one issue by adding appropriate line reflecting what is done in 'maps{ii} = mymap' line. It definitely helped, but there are still some differences.

I'm afraid that building the objects directly in the parfor loop would not work here, as there's lot happening before parfor and also it is wrapped in an outside while loop; perhaps it would bring too much overhead than benefit of using. Although, I managed to read the A(i).cpx.Solution.x values into the cell array and assign them to A(i).x outside parfor loop, so I think there is no other way than just read all the others in the same manner.

Thank you Edric for quick response!