Reinforcement Learning Toolbox: Not enough Room in buffer

Problem was a missunderstanding of an example. So this error is caused by an user error that was explained in one of the comments to this issue if that comment does not apply to you, you most likely have a different issue.

Hello,

I am pretty new in the realm of RL and am using the RL Toolbox to controll a Simulink modell with the DDPG Agent.

I have 2 actions and 2 observations

My Problem is that everytime i try to train the agent I get the Error:

An error occurred while running the simulation and the simulation was terminated

Caused by:

MATLAB System block 'rlMockLoop/RL Agent/AgentWrapper' error occurred when invoking 'outputImpl' method of 'AgentWrapper'. The error was thrown from '

'/usr/local/MATLAB/R2019a/toolbox/rl/rl/+rl/+util/ExperienceLogger.m' at line 30

'/usr/local/MATLAB/R2019a/toolbox/rl/rl/+rl/+agent/AbstractPolicy.m' at line 95

'/usr/local/MATLAB/R2019a/toolbox/rl/rl/simulink/libs/AgentWrapper.m' at line 107'.

Not enough room in the buffer to store the new experiences. Make sure the bufferSize argument is big enough.

I tried to increase the agentOption ExperienceBufferLength (even to pretty high values).

Is that even the right Option I should be looking at or am I missing something?

Code snippets:

Ts ~ 0.05
actionInfo = rlNumericSpec([2 1],...
    'LowerLimit',[0 0]',...
    'UpperLimit',[100  100]');
actionInfo.Name = 'StromstaerkeProzent';
actionInfo.Description = 'Aout, Ain';

%% Specify Observations
observationInfo = rlNumericSpec([2 1]);
actionInfo.Name = 'pressure';
actionInfo.Description = 'DruckWasser, Druck';

agentOpts = rlDDPGAgentOptions(...
    'SampleTime',Ts,...
    'TargetSmoothFactor',1e-3,...
    'ExperienceBufferLength',512*((10/Ts)*1000),...
    'DiscountFactor',0.99,...
    'MiniBatchSize',512);

agent = rlDDPGAgent(actor,critic,agentOpts);

trainingOptions = rlTrainingOptions(...
    'MaxEpisodes',1000, ...
    'MaxStepsPerEpisode',10/Ts, ...
    'ScoreAveragingWindowLength',5,...
    'Verbose',false, ...
    'Plots','training-progress',...
    'StopTrainingCriteria','AverageReward',...
    'StopTrainingValue',-1100,...
    'SaveAgentCriteria','EpisodeReward',...
    'SaveAgentValue',-1100);

simOptions = rlSimulationOptions('MaxSteps',10/Ts);
experience = sim(env,agent,simOptions);

Other:

I tried to make the buffer size relative to the episode count and the length of 10s.

I really hope somebody can help me.

[Emmanouil Tzorakoleftherakis]

Hi Clemens,

Can you share a repro model? I may be able to help if I can reproduce the error.

[Clemens Fricke]

I am sorry I missunderstood a part of the example and used the sim() command instead of the train() command. And used an untrained agent during the simulation. I think i had something todo with that.

With train() I do not get that error anymore.

The Modell still crashes after ca 20 h training. This error is I think a different one than the one I had while using the wrong command.

Should I open a different Ticket for the new Problem?

Where should I share the Model?

[Anh Tran]

Feel free to open a new question since the new issue is not related to the original.

[Saurav Sthapit]

Did anyone solve this issue?

I am facing the same isue in R2019a when simulatng with sim(env,agent,simopts) after training the agent.

[eyman ikhlaq]

i am facing the same issue please send a solution

[Aysegul Kahraman]

Hi,

You can try to use delay block after the action.

It solved my problem although that was not my first choice to solve this issue.

Answer 1

[YUCHEN Liu]

Sometimes connecting the output of the agent directly back to the reward will cause this situation, maybe you need a delay module

[Aysegul Kahraman]

I am facing the same issue and I could not find a solution. There is definitely a relationship to not having a delay block. However, I have a delay block before the reward which should be enough.

Any solution or suggestion?