Deep Reinforcement Learning Toolbox with LSTM RNN

5 views (last 30 days)
Hi,
Currently I am using a LSTM RNN network in my TD3 agent model.
I am using a LSTM network architecture based on the DDPG TD3 the online help, as shown below
%Critic Network
rng(0) % fix the random seed
statePath1 = [
sequenceInputLayer(numObservations,'Normalization','none','Name','observation')
fullyConnectedLayer(400,'Name','CriticStateFC1')
reluLayer('Name','CriticStateRelu1')
fullyConnectedLayer(300,'Name','CriticStateFC2')
];
actionPath1 = [
sequenceInputLayer(numActions,'Normalization','none','Name','action')
fullyConnectedLayer(300,'Name','CriticActionFC1')
];
commonPath1 = [
additionLayer(2,'Name','add')
reluLayer('Name','CriticCommonRelu1')
lstmLayer(16,'OutputMode','sequence','Name','CriticLSTM');
fullyConnectedLayer(1,'Name','CriticOutput')
];
criticNet = layerGraph(statePath1);
criticNet = addLayers(criticNet,actionPath1);
criticNet = addLayers(criticNet,commonPath1);
criticNet = connectLayers(criticNet,'CriticStateFC2','add/in1');
criticNet = connectLayers(criticNet,'CriticActionFC1','add/in2');
%Actor Network
actorNet = [
sequenceInputLayer(numObservations,'Normalization','none','Name','observation')
fullyConnectedLayer(400,'Name','ActorFC1')
lstmLayer(8,'OutputMode','sequence','Name','ActorLSTM')
reluLayer('Name','ActorRelu1')
fullyConnectedLayer(300,'Name','ActorFC2')
reluLayer('Name','ActorRelu2')
fullyConnectedLayer(numActions,'Name','ActorFC3')
tanhLayer('Name','ActorTanh1')
];
My Mini Batch size is set to 200.
When my training begins the model runs very quickly, during the initial episodes, which is the same as the mini batch size of 200, then from episode 201 onward the model runs at a normal training pace and seems to start learning, this can be seen in the episode manager plot below.
My question is:
Is this initial training behavour normal for a LSTM RNN?
Cheers
Patrick

Answers (0)

Products


Release

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!