Parallel workers automatically shutting down in the middle of RL parallel training.
21 views (last 30 days)
Show older comments
Matteo D'Ambrosio
on 10 May 2023
Commented: Matteo D'Ambrosio
on 11 May 2023
Hello,
I am currently training a reinforcement learning PPO agent on a Simulink model with UseParallel=true. The total episodes that it should be running for are 5000 (about 10/11 hours of training), but i'm noticing that as the training goes on, more and more workers of the parallel pool are automatically shutting down, making training slower and slower as it progresses. I start with 8 workers, and they consistently decrease one at a time, until errors are generated.
I've been noticing this consistently in each training that i do, and would like to know if there are any workarounds.
For the parpool, i am letting Matlab start it automatically with all options set to default. I have also tried playing around with the number of workers, but the same thing happens.
2 Comments
Emmanouil Tzorakoleftherakis
on 10 May 2023
What errors are you seeing? Maybe try training on a single worker initially to make sure you don't see any errors before moving to parallel.
Accepted Answer
Edric Ellis
on 11 May 2023
If workers are leaving the pool one at a time while the pool is busy, this almost certainly means that they are crashing. I recommend contacting MathWorks support for help diagnosing and resolving this problem. You could also check by running the following location to see if any "matlab_crash_dump.*" files have been left behind:
jsl = parcluster().JobStorageLocation
dir(fullfile(jsl, '**', 'matlab_crash_dump.*'))
More Answers (0)
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!