MATLAB Answers

Train a DQN Agent: RL Simulink with Simscape (Error: Discontinuities detected within algebraic loop)

3 views (last 30 days)
Reinforcement Learning
Reinforcement Learning on 18 Mar 2021
Hello,
I am trying to replace the PI-Controller (the highlighted area with PWM Generator of 50 kHz) for the Buck-Converter sketch with A DQN-Agent!
Here the Simulink Enviroment with the PI-Controller, which is performing well
I tried to replicate the MATLAB example "Water Tank Model" in Simulink using RL, but using DQN instead of DDPG (since the action is discrete) and replaced the environment with the buck-converter with some other minor changes (tolerance ), but I am getting various errors, when I use validateEnvironment. Here my Matlab code with the simulink environment:
Obs_Info = rlNumericSpec([1 1]);
Obs_Info.Name = 'observations'; % measured current
numObservations = Obs_Info.Dimension(1);
% specifies discrete action specifications
Act_Info = rlFiniteSetSpec([0 1]);
Act_Info.Name = 'PWM';
numActions = Act_Info.Dimension(1);
env = rlSimulinkEnv('rl_converter','rl_converter/RL Agent', Obs_Info, Act_Info);
workspace = 'Stromsteuerung_rl';
env.ResetFcn = @(in)setVariable(in,'observations',0,'Workspace',workspace);
% the agent gets executed every SampleTime seconds of simulation time
Ts = 1/50000; % (50 kHz)
Tf = (1/50000)*60; % simulation time
%% Deep Neural Network
dnn = [
featureInputLayer(1,'Normalization','none','Name','State')
fullyConnectedLayer(40,'Name','CriticStateFC1')
reluLayer('Name','CriticRelu1')
fullyConnectedLayer(40, 'Name','CriticStateFC2')
reluLayer('Name','CriticCommonRelu')
fullyConnectedLayer(2,'Name','Action')];
%% Representation of Q-Values
Critic_Opts = rlRepresentationOptions;
Critic_Opts.LearnRate = 0.001;
Critic_Opts.GradientThreshold = 1;
critic = rlQValueRepresentation(dnn, Obs_Info, Act_Info, 'Observation',{'State'},Critic_Opts);
%% Agent options
Ag_Opts = rlDQNAgentOptions;
Ag_Opts.UseDoubleDQN = true;
Ag_Opts.TargetSmoothFactor = 1;
Ag_Opts.TargetUpdateFrequency = 4;
Ag_Opts.ExperienceBufferLength = 100000;
Ag_Opts.DiscountFactor = 0.9;
Ag_Opts.MiniBatchSize = 64;
Ag_Opts.SampleTime = Ts;
Ag_Opts.EpsilonGreedyExploration.Epsilon = 0.5;
agent = rlDQNAgent(critic,Ag_Opts);
%% Training options
Train_Opts = rlTrainingOptions;
Train_Opts.MaxEpisodes = 100;
Train_Opts.MaxStepsPerEpisode = ceil(Tf/Ts);
Train_Opts.StopTrainingCriteria = "AverageReward"; % or AverageSteps
Train_Opts.StopTrainingValue = 800;
Train_Opts.Verbose = false;
Train_Opts.Plots = "training-progress";
%% Validate
validateEnvironment(env)
Warning: Discontinuities detected within algebraic loop(s), may have trouble solving
Warning: Convergence problem when solving algebraic loop containing 'rl_converter/stop simulation/Compare To Constant1/Compare'
at time 0.0. Simulink will try to solve this loop using Simulink 3 (R11) strategy.
Use feature('ModeIterationsInAlgLoops',0) to disable the strategy introduced in Simulink 4 (R12)
Why isn't this warning showing in the Water Tank Model and only in this one? Is Simscape not compatible with this application. If so, how could I change my environment to something that goes well with the RL Toolbox?
Any help is very much appreciated!

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!