Train a DQN Agent: RL Simulink with Simscape (Error: Discontinuities detected within algebraic loop)

Hello,

I am trying to replace the PI-Controller (the highlighted area with PWM Generator of 50 kHz) for the Buck-Converter sketch with A DQN-Agent!

Here the Simulink Enviroment with the PI-Controller, which is performing well

I tried to replicate the MATLAB example "Water Tank Model" in Simulink using RL, but using DQN instead of DDPG (since the action is discrete) and replaced the environment with the buck-converter with some other minor changes (tolerance ), but I am getting various errors, when I use validateEnvironment. Here my Matlab code with the simulink environment:

Obs_Info = rlNumericSpec([1 1]);
Obs_Info.Name = 'observations'; % measured current
numObservations = Obs_Info.Dimension(1);

% specifies discrete action specifications
Act_Info = rlFiniteSetSpec([0 1]);
Act_Info.Name = 'PWM';
numActions = Act_Info.Dimension(1);

env = rlSimulinkEnv('rl_converter','rl_converter/RL Agent', Obs_Info, Act_Info);

workspace = 'Stromsteuerung_rl';
env.ResetFcn = @(in)setVariable(in,'observations',0,'Workspace',workspace);

% the agent gets executed every SampleTime seconds of simulation time

Ts = 1/50000;        % (50 kHz)
Tf = (1/50000)*60;   % simulation time

%% Deep Neural Network
dnn = [
    featureInputLayer(1,'Normalization','none','Name','State')
    fullyConnectedLayer(40,'Name','CriticStateFC1')
    reluLayer('Name','CriticRelu1')
    fullyConnectedLayer(40, 'Name','CriticStateFC2')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(2,'Name','Action')];

%% Representation of Q-Values
Critic_Opts = rlRepresentationOptions;
Critic_Opts.LearnRate = 0.001;
Critic_Opts.GradientThreshold = 1;

critic = rlQValueRepresentation(dnn, Obs_Info, Act_Info, 'Observation',{'State'},Critic_Opts);

%% Agent options
Ag_Opts = rlDQNAgentOptions;

Ag_Opts.UseDoubleDQN = true;
Ag_Opts.TargetSmoothFactor = 1;
Ag_Opts.TargetUpdateFrequency = 4;
Ag_Opts.ExperienceBufferLength = 100000;
Ag_Opts.DiscountFactor = 0.9;
Ag_Opts.MiniBatchSize = 64;
Ag_Opts.SampleTime = Ts;

Ag_Opts.EpsilonGreedyExploration.Epsilon = 0.5;

agent = rlDQNAgent(critic,Ag_Opts);

%% Training options

Train_Opts = rlTrainingOptions;

Train_Opts.MaxEpisodes = 100;
Train_Opts.MaxStepsPerEpisode = ceil(Tf/Ts);
Train_Opts.StopTrainingCriteria = "AverageReward"; % or AverageSteps
Train_Opts.StopTrainingValue = 800; 
Train_Opts.Verbose = false;
Train_Opts.Plots = "training-progress";

%% Validate

validateEnvironment(env)

Warning: Discontinuities detected within algebraic loop(s), may have trouble solving

Warning: Convergence problem when solving algebraic loop containing 'rl_converter/stop simulation/Compare To Constant1/Compare'

at time 0.0. Simulink will try to solve this loop using Simulink 3 (R11) strategy.

Use feature('ModeIterationsInAlgLoops',0) to disable the strategy introduced in Simulink 4 (R12)

Why isn't this warning showing in the Water Tank Model and only in this one? Is Simscape not compatible with this application. If so, how could I change my environment to something that goes well with the RL Toolbox?

Any help is very much appreciated!

[Francisco Serra]

Hey! Could you solve it?