Reinforcement Learning: Use trained agent in Simulink

9 views (last 30 days)
Hello,
I have a problem to use my trained agent in Simulink. I try to structure this thread so good as possible. First of all I want to mention that I use Matlab 2020b.
At first I want to introduce the structure of my neural network:
mdl = 'Simulink_Model' % Speichern des Simulink-Modells in der Variable mdl
% Actions
actionNum = 1 % Anzahl der Aktionen
actionInfo = rlNumericSpec([1 1],... % Erstellung des Action Spezifikations Objekts
'LowerLimit', -9,...
'UpperLimit', 9);
actionInfo.Name = 'Voltage Supply'; % Bezeichnung des Action Spezifikations Objekts
% Observations
observationNum = 4 % Anzahl der Beobachtungen
observationInfo = rlNumericSpec([4 1],... % Erstellung des Observation Spezifikations Objekts
'LowerLimit', [-inf -inf -pi()/2 -inf]',...
'UpperLimit', [inf inf pi()/2 inf]');
observationInfo.Name = 'Observations';
observationInfo.Description = 'Wheel Velocity, Wheel Position, Body Angle, Angle Velocity'; % Bezeichnung des Oberservation Spezifikations Objekts
% Environment
env = rlSimulinkEnv(mdl, 'Simulink_Model/RL Agent', observationInfo, actionInfo)
env.ResetFcn = @(in)localResetFunction(in) % Reset Function
rng(0) % Fix the random generator seed for reproducibility.
% Neural Network of Agent
statePath = [
featureInputLayer(observationNum,'Normalization','none','Name','observation')
fullyConnectedLayer(400,'Name','CriticStateFC1')
reluLayer('Name', 'CriticRelu1')
fullyConnectedLayer(300,'Name','CriticStateFC2')];
actionPath = [
featureInputLayer(1,'Normalization','none','Name','action')
fullyConnectedLayer(300,'Name','CriticActionFC1','BiasLearnRateFactor',0)];
commonPath = [
additionLayer(2,'Name','add')
reluLayer('Name','CriticCommonRelu')
fullyConnectedLayer(1,'Name','CriticOutput')];
criticNetwork = layerGraph();
criticNetwork = addLayers(criticNetwork,statePath);
criticNetwork = addLayers(criticNetwork,actionPath);
criticNetwork = addLayers(criticNetwork,commonPath);
criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');
criticNetwork = connectLayers(criticNetwork,'CriticActionFC1','add/in2');
criticOptions = rlRepresentationOptions('LearnRate',2e-03,'GradientThreshold',1);
% Critic
critic = rlQValueRepresentation(criticNetwork,observationInfo,actionInfo,'Observation',{'observation'},'Action',{'action'},criticOptions);
% Actor
actorNetwork = [
featureInputLayer(observationNum,'Normalization','none','Name','observation')
fullyConnectedLayer(400,'Name','ActorFC1')
reluLayer('Name','ActorRelu1')
fullyConnectedLayer(300,'Name','ActorFC2')
reluLayer('Name','ActorRelu2')
fullyConnectedLayer(1,'Name','ActorFC3')
tanhLayer('Name','ActorTanh')
scalingLayer('Name','ActorScaling','Scale',max(actionInfo.UpperLimit))];
actorOpts = rlRepresentationOptions('LearnRate',1e-04,'GradientThreshold',1);
actor = rlDeterministicActorRepresentation(actorNetwork,observationInfo,actionInfo,'Observation',{'observation'},'Action',{'ActorScaling'},actorOpts);
% Agent
Ts = str2num(get_param(mdl, 'FixedStep')) % Sample Time des Simulink Modells
agentOptions = rlDDPGAgentOptions(...
'SampleTime',Ts,...
'TargetSmoothFactor',1e-1,...
'ExperienceBufferLength',1e6 ,...
'DiscountFactor',0.99,...
'MiniBatchSize',256);
agentOptions.NoiseOptions.Variance = 0.6;
agentOptions.NoiseOptions.VarianceDecayRate = 1e-5
% Creating the Agent
agent_DDPG = rlDDPGAgent(actor,critic,agentOptions)
After I trained the agent, I have wanted to realize the trained agent in the real world. I have read that I have to use a matlab function instead of my RL Agent in Simulink here. So I generated with 'generatePolicyFunction' a function evaluatePolicy.m, that is structured liked that:
function action1 = evaluatePolicy(observation1)
%#codegen
% Reinforcement Learning Toolbox
% Generated on: 10-Dec-2021 12:48:51
action1 = localEvaluate(observation1);
end
%% Local Functions
function action1 = localEvaluate(observation1)
persistent policy
if isempty(policy)
policy = coder.loadDeepLearningNetwork('agentData.mat','policy');
end
observation1 = reshape(observation1,[1 1 4]);
action1 = predict(policy,observation1);
end
My Simulink looks like that:
An the MATLAB function is structured like that:
function y = fcn(u)
ausgabe = evaluatePolicy(u)
y = ausgabe
end
If I try 'Monitor and Tune', I get the following error code:
Invalid value set for 'DLTargetLibrary' for deep learning code generation. The allowed values are: 'MKL-DNN' , 'ARM Compute'.
Function 'evaluatePolicy.m' (#49.286.341), line 13, column 11:
"coder.loadDeepLearningNetwork('agentData.mat','policy')"
Component:MATLAB Function | Category:Coder error
Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment.
Function 'evaluatePolicy.m' (#49.411.417), line 16, column 19:
"policy"
Component:MATLAB Function | Category:Coder error
Function call failed.
Function 'evaluatePolicy.m' (#49.140.167), line 7, column 11:
"localEvaluate(observation1)"
Component:MATLAB Function | Category:Coder error
Function call failed.
Function 'MATLAB Function' (#36.34.51), line 2, column 15:
"evaluatePolicy(u)"
Component:MATLAB Function | Category:Coder error
Undefined function or variable 'ausgabe'. The first assignment to a local variable determines its class.
Function 'MATLAB Function' (#36.60.67), line 3, column 9:
"ausgabe"
Component:MATLAB Function | Category:Coder error
Errors occurred during parsing of MATLAB function 'Real_Model/MATLAB Function'
In my opinion this error doesn't make any sense, because I have set the value for DLTargetLibrary at 'MKL-DNN' as you can see here:
I would really appreciate it if somebody can help me with that issue.
  1 Comment
Filip Poloczek
Filip Poloczek on 17 Dec 2021
I have also followed the instruction here to build the Intel MKL-DNN library. Unfortunately I get the same error code.

Sign in to comment.

Answers (1)

Sayan Saha
Sayan Saha on 22 Dec 2021
Hi Filip,
Can you check if you have selected "MKL-DNN" as the Deep Learning Target Libray under "Interface" in "Code Generation" pane? See below image for reference:
The option that you have selected currently is only for the simulation mode. For code-generation the target library needs to be set too as mentioned above and these settings for simulation and code-generation are independent of each other.
Thanks,
Sayan
  7 Comments
Filip Poloczek
Filip Poloczek on 6 Jan 2022
Edited: Filip Poloczek on 6 Jan 2022
Hi Sayan,
I would like to use Matlab R2021a so I deleted the ScalingLayer and trained my agent from beginning. In the simulation my agent worked well, so I wanted to use it now in external mode. But this time I get the following error code:
Starting build procedure for: Real_Model
Maximum variable size allowed by the program is exceeded. Function 'evaluatePolicy.m' (#92.397.426), line 16, column 18: "predict(policy, observation1)"
Component: MATLAB Function | Category: Coder error
Function call failed. Function 'evaluatePolicy.m' (#92.140.167), line 7, column 11: "localEvaluate(observation1)"
Component: MATLAB Function | Category: Coder error
Function call failed. Function 'MATLAB Function' (#67.35.52), line 2, column 15: "evaluatePolicy(u)"
Component: MATLAB Function | Category: Coder error
Undefined function or variable 'ausgabe'. The first assignment to a local variable determines its class. Function 'MATLAB Function' (#67.61.68), line 3, column 9: "ausgabe"
Component: MATLAB Function | Category: Coder error
Errors occurred during parsing of MATLAB function 'Real_Model/MATLAB Function'
Component: MATLAB Function | Category: Coder error
I couldn't find any answer how I can solve this problem. Do you now maybe a way to solve that problem?
Sincerely,
Filip
Sayan Saha
Sayan Saha on 6 Jan 2022
Hi Filip,
From the error message it seems like the RL model might be too large to fit in the memory available in the board. What is the model architecture? Can you try deploying a very simple model to check if that generates code and deploys to the board correctly?
~Sayan

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!