What happens if I use parfor and the fmincon/fminunc option "UseParallel" together?

I want to minimize the same objective function using n different starting points. I have k cores available. I have .

Consider the following (toy) code:

parfor ii = 1:n
    x_opt(ii) = fminunc( obj_fun, starting_point(ii), 'UseParallel', true )
end

Is MatLab smart enough to allocate cores to each optimization?

[Mario Malic]

Hello,

While someone else with proper knowledge comes, I'd take a guess (as there isn't so much information about 'UseParallel').

If your number of cores is higher by a factor of two, you can try setting that number as "computational threads" (physical cores?) each worker uses, see Cluster Profile Manager.

From the documentation on parfor, there is a syntax on specifying maximum number of edited: MATLAB processes M to use, maybe it'll somehow distribute more threads to each worker (default is 1 worker - 1 thread) that way.

parfor (loopVar = initVal:endVal,M); statements; end

[Raymond Norris]

@Michael, regarding cores per optimization, are you asking that if you have 8 cores and 5 sims, will MATLAB allocate1.60 cores per sim?

@Mario, slight clarification. M is the number of MATLAB processes, not threads, to use. The way to increase the number of computation threads for each worker is to set the value in the cluster object. For example:

c = parcluster;
c.NumThreads = 3;
p = c.parpool(4);

This will start 4 workers on 4 cores and allocate two additional cores per worker for additional threads for each worker.

[Matt J]

I think the question relates to how resources are divided up when you have nested parfor loops. The function fminunc internally calls parfor when the UseParallel option is invoked, and therefore we effectively have nested parfor loops. If an outer parfor worker itself launches an inner parfor (indirectly through a function call), will the inner parfor divide work across the entire parpool, or will it be confined to whatever resources are assigned to the outer worker that invoked it?

[Michael Stollenwerk]

@Raymond yes, to be specific I have 32 cores available and 4 sims. Obviously I am not asking about fractional cores (like 1.60).

[Raymond Norris]

With an indirect nested parfor loop, the inner parfor will operate essentially as a for loop. An explicit nested parfor loop will error out.

[Michael Stollenwerk]

Ok, thanks. That's unfortuante, since I have the suspicion that in my case 'UseParallel' with corse would improve speed by a lot, since obj_fun has a high-dimensional input array and thus many (parallel) function evaluations are used to estimate the gradient in each step.

[Mario Malic]

I think it's worthy to try a serial optimisation for outer loop and UseParallel option, especially if your function evaluation takes some time to evaluate.

Answer 1

[Matt J]

I don't know the answer to that directly, but it is easy to show by example that turning off 'UseParallel' can be beneficial:

objfun=@(x)sum(x.^2);
x0=rand(2000,1);
opts=optimoptions('fminunc','Display','none','UseParallel',true);

tic;
parfor i=1:4
    
    fminunc(objfun,x0,opts);
end
toc%Elapsed time is 0.369885 seconds.

opts.UseParallel=false;
tic;
parfor i=1:4
    
    fminunc(objfun,x0,opts);
end
toc%Elapsed time is 0.274740 seconds.

[Michael Stollenwerk]

Thanks for your answer. Do you have more than 4 cores available?

[Matt J]

Yes, I have 12.

 ProcessPool with properties: 

            Connected: true
           NumWorkers: 12
              Cluster: local
        AttachedFiles: {}
    AutoAddClientPath: true
          IdleTimeout: 30 minutes (21 minutes remaining)
          SpmdEnabled: true

[Michael Stollenwerk]

Thanks!

[Mario Malic]

Can you edit your numThreads per worker and try again?

[Matt J]

Edit it to what?

[Mario Malic]

Sorry for missing out on this information, 4 workers, 3 threads per worker.

Actually, timing might be even worse, because time to set up problem to solve in parallel is higher than actual solving time.

[Matt J]

OK, here's what I did:

objfun=@(x)sum(x.^2);
x0=rand(2000,1);
opts=optimoptions('fminunc','Display','none','UseParallel',true);

tic;
parfor (i=1:20,4)
    
    fminunc(objfun,x0,opts);
end
toc%Elapsed time is 1.410020 seconds.



opts.UseParallel=false;
tic;
parfor (i=1:20,4)
    
    fminunc(objfun,x0,opts);
end
toc%Elapsed time is 0.995813 seconds.

[Raymond Norris]

I'm puzzled how Matt's example worked when UseParallel is set to true. If you're running a parfor loop, I would think the optimization would not use the parallel pool and hence run serially. This would explain slightly why the performance.

Secondly, 2000x1 most likely just isn't big enough. Take a look at the following with 20000. For starters, when I benchmark, I'm setting maxNumCompThreads to 1 for a baseline and the setting to 4 later. This tells me what implicit multi-threading MATLAB gives me.

objfun = @(x)sum(x.^2);
x0 = rand(20000,1);

opts = optimoptions('fminunc','Display','none','UseParallel',false);

%% Baseline
maxNumCompThreads(1);
tic
for i=1:4
    fminunc(objfun,x0,opts);
end
toc
% Elapsed time is 197.351202 seconds.

%% 4 threads, no additional parallelism
maxNumCompThreads(4);
tic
for i=1:4
    fminunc(objfun,x0,opts);
end
toc
% Elapsed time is 75.309341 seconds.

%% 4 workers, but parallelism is minimal 
opts = optimoptions('fminunc','Display','none','UseParallel',true);
tic
for i=1:4
    fminunc(objfun,x0,opts);
end
toc
% Elapsed time is 64.789117 seconds.

[Matt J]

I'm not sure what the manipulation of maxNumCompThreads is supposed to tell us about the interaction between parfor and UseParallel. I would think that reducing maxNumCompThreads will add even more bottlenecks to the computation that wouldn't otherwise be there, because now fminunc cannot maximally exploit the multi-core resources even for basic linear algebra steps. With more bottlenecks, the parallelization or non-parallelization of the gradient calculation step (controlled by UseParallel) will have a diminished impact.