Specify the parallel pool job timeout

9 views (last 30 days)
Maria
Maria on 1 Oct 2021
Commented: Raymond Norris on 2 Oct 2021
Hi,
I am running some tests using a remote cluster (no local cluster). I send my functions in batch mode. I know that the functions take a large execution time, around 2 / 4 hours. When I try to run, I get the message:
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I looked into the documentation how to change the default timeout time. The only way I could find is with the "wait" command, as
wait(job,"finished",18000);
However, I keep on getting the same error. How can I change the default parallel pool job timeout in the remote cluster?

Answers (1)

Raymond Norris
Raymond Norris on 1 Oct 2021
So you doing something like the following
cluster = parcluster;
job = cluster.batch(@mycode,...., 'Pool',size);
Then what you're suggesting is that let's say your code looked something like
function mycode
pause(10 * 60)
parfor idx = 1:N
...
end
On the cluster, the workers have a default timeout of 5 minutes and therefore errors out because you're running code (the pause) for 10 minutes before the workers are being used in the parfor.
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I tried quickly reproducing this with the local scheduler, but didn't (it shouldn't matter if I'm jusing local). How are you getting the error message? And which scheduler are you using MJS or generic (e.g. PBS)?
  9 Comments
Raymond Norris
Raymond Norris on 2 Oct 2021
One of the workers crashed, possibly because of an out of memory issue. Email Technical Support (support@mathworks.com) and they can walk you through debug steps and getting the MJS log files to troubleshoot.

Sign in to comment.

Products


Release

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!