Matlab 2024b crashes when trying to continue training an RL agent

Hello together,

I have trained an TD3 agent in Matlab with Simulink environment. I saved the trained agent and experienceBuffer in a mat-file (it is around 9.8GB). If I load the agent and do the simulation, I can see how the agent is acting, so I think everything is fine so far.

simOpts = rlSimulationOptions(MaxSteps=ceil(RL.Tf/RL.Ts), StopOnError="on", SimulationStorageType='memory'); %file
experiences = sim(env,agent,simOpts);

Now I would like to continue training the agent and there I run into a problem.

trainingStats  = trainWithEvolutionStrategy(agent,env,evsTrainingOpts);
%trainingStats = train(agent,env,trainOpts);
agent.AgentOptions.SaveExperienceBufferWithAgent = true;
folderName_for_save = folderName;
save(fullfile(folderName_for_save, 'trainedAgent.mat'), 'agent', '-v7.3');
save(fullfile(folderName_for_save, 'trainingStats.mat'), 'trainingStats', '-v7.3');

I do the training in parallel with 8 workers and it will not start again. I get this error:

Is it because I only have an AMD Ryzon 5000 with only 16GB DDR4 RAM and I would need 32GB RAM? Or is it something else?

Alex

[Sumukh]

Hi @Alexander Enzmann,

The exit status does seem to indicate that this is an out-of-memory issue.

Can you once try increasing the Java Heap Memory size at Preferences -> MATLAB -> General -> Java Heap Memory and see if this issue persists?

If it still crashes, please contact MathWorks Support for better assistance in resolving the crash:

https://www.mathworks.com/support/contact_us.html

Answer 1

[Alexander]

thank you. I tried different settings for Java Heap Memory but error still occurs. Right now I go with a little workaround. I load the trained agent into workspace and extract the trained weights for actor and critic1 and critic2 (but I dont use the experienceBuffer). Then I create a new agent and load the weights from the trained agent. Before starting training again I delete the old agent from workspace (to get free RAM).

Now I can start the training using the weights from the trained agent, but I have to go with a new experienceBuffer.