Parfor number of workers, CPUs, how are they related? What are the limits?

29 views (last 30 days)
Using parfor specifying the number of parallel workers, how does it relate to the number of CPUs?
Where M specifies the Maximum number of workers running in parallel, as both a cap, and a limit.
It may seem like a very obvious question and answer: is the number of workers equal to the CPUs available? That is, on a quadcore computer, the maximum number of workers would be 4?
A separate question, I've been advised I can set some settings using this written in somewhere before the parfor coding:
tempdir = getenv('TMPDIR')
c = parcluster
c.JobStorageLocation = tempdir
Would this specify the pool to 8 workers? In which case the code below for parfor would cap at 8 even if M is specifed as 10? What would happen if M was 4? Would it be capped at 4 or 8?
parfor (i=1:10,10)
parfor (i=1:10,4)
And where should theTMPDIR in tempdir = getenv('TMPDIR') be located?
Thanks a lot!

Accepted Answer

Raymond Norris
Raymond Norris on 30 Jun 2021
A quadcore computer should max at 4 workers. We don't suggest throttling it to include hyperthreads. With that said, let's say you have 4 additional HT and you start a pool of 8 workers, perhaps you get even a 10% increase (for example), then maybe it's worth it. You just shouldn't expect an 8x improvement.
Setting the JobStorageLocation is only a suggestion if you are running MATLAB jobs on an HPC cluster and in some sort of rapid batch mode. For example, you write a PBS job script to run MATLAB code that uses a local pool. And then you submit a slew of those at once. In that case, setting the JSL can be helpful (due to a race condition).
The number of workers used is the min(M,size-of-pool). In your example, that would be 8 and 4 respectfully.

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!