I'm starting to think this has to do with what parfor does, which is calling simply calling omp. Isn't something more needed than omp for multiple nodes, like actual mpi calls? In that case, it seems matlab is incapable of running on more than 1 node, even with parallel server, or does PS somehow extend parfor (and thus omp)?
Cluster uses only one node, even though 5 nodes Running (parfor)
9 views (last 30 days)
Show older comments
Commented: Raymond Norris on 17 Oct 2022
I am running on our cluster an .m file which contains parfor loops.
Using a cluster config file (below), I request 5 nodes, each nodes has 48 procs, which I open with parpool(5*48) workers.
I am able to logon to each node individually and monitor its use with htop.
I see that on each node indeed matlab is running. But it seems that only 1 node is being used by looking at ram usage. If I increase the parfor loop, the process crashes.
I was looking at other posts, that mention setting a parameter LASTN = maxNumCompThreads('automatic'). This is equal to 30 before setting anything. I have tried setting this equal to the total number of workers, eg numProcs*numNodes=48*5.
- Here is the node that's being used: 111-150GB being used. When the parfor loop is too big, that node's ram maxes out at 187 and matlab crashes. Also, perhaps noteworthy this is not the head node.
2. Here is an example of an unused node: Note that 34 G is the total ram usage when matlab is idle
This is the config file I run, before I run my job. To confirm stuff before the run I can do eg c.AdditionalProperties.NumNodes and ... .numWorkers and get the expected amount (numnodes=numworkers=5*48)
allNames = parallel.clusterProfiles()
c.AdditionalProperties.WallTime = '24:20:0';
c.AdditionalProperties.QueueName = 'CAC48M192_L';
c.AdditionalProperties.AccountName = 'redacted';
nn = 5;
pp = 48;
c.AdditionalProperties.NumNodes = nn;
c.AdditionalProperties.ProcsPerNode = pp;
c.AdditionalProperties.NumWorkers = pp*nn;
More Answers (1)
Raymond Norris on 6 Oct 2022
Starting a parpool will create very little activity for the workers. It's only once you run a parallel construct (e.g., parfor) will any work be given to the workers.
Increasing the parfor loop shouldn't crash the workers. What are working are you doing? Did you request enough memory (c.AdditionalProperties.MemUsage)? Or do you mean increasing the parallel pool?
I can't tell where you're running MATLAB (your machine, head node, compute node?), but I suspect if the maxNumCompThreads is 30, and there are 48 cores per node, you're not running MATLAB on the compute node. You don't want thread count to be greater than the number of cores on a node (since they won't spawn across nodes). To set the thread count, call
c.NumThreads = 48;
This will then force each worker to run on its own node.
Workers are capable of running more than one node. Post a bit more context and we can figure out what's going on.
- Are you running MATLAB on your machine, head node, or compute node?
- Are you sbumitting jobs with parpool or batch?
- What size pool are you starting (i.e., parpool(X) or batch(.., 'Pool', X);
- Can you provide the code you're trying to run?
Raymond Norris on 17 Oct 2022
Slurm is telling you that your combonation of cores, threads, and nodes is not available. By the looks of it, I'd say someone on our team helped implement when you're doing (i.e., using configCluster). If you're interested, contact Technical Support (email@example.com) with your contact info. They in turn can get a hold of me. We can work this out offline.
Find more on Parallel Computing Fundamentals in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!