Clear Filters
Clear Filters

Definition of transitions and rewards in a MDP environment for a reinforcement learning problem.

7 views (last 30 days)
Good morning,
I have an environment with 50 states and 5 actions. From state 1, I transition to states 2 to 25 with action 1; from state 1, I transition to states 26 to 49 with action 2; from states 2 to 25, I transition to state 50 with action 3; from states 26 to 49, I transition to state 50 with action 4. I define the matrices MDP.T as follows:
MDP.T(1,2:25,1) = 1/24;
MDP.T(1,26:49,2) = 1/24;
MDP.T(2:25,50,3) = 1;
MDP.T(26:49,50,4) = 1;
MDP.T(50,50,5) = 0;
The code works only if I consider also these other transitions, which are not feasible in my example.
MDP.T(1,1,3) = 1;
MDP.T(1,1,4) = 1;
MDP.T(2:25,2:25,1) = 1/24;
MDP.T(2:25,2:25,2) = 1/24;
MDP.T(2:25,2:25,4) = 1/24;
MDP.T(2:25,2:25,5) = 1/24;
MDP.T(26:49,26:49,1) = 1/24;
MDP.T(26:49,26:49,2) = 1/24;
MDP.T(26:49,26:49,3) = 1/24;
MDP.T(26:49,26:49,5) = 1/24;
May you help me to understand how to define only feabile transitions?
Thank you in advance
  3 Comments
Bianca Grieco
Bianca Grieco on 12 Jan 2024
Dear Sir, I am implementing a reinforcement learning method in MATLAB for my thesis. Prior to my previous message, I am writing to ask if you could provide me with information regarding both the maximum dimensions of each MDP.T matrix and the total number of MDP.T matrices (and consequently actions) that can be handled by the MATLAB RL toolbox. Thank you in advance for your response.
Bianca Grieco
Bianca Grieco on 14 Jan 2024
Dear Sir Tzorakoleftherakis, have you had time to check the answer respect to the maximum dimension of the transition matrix? Thank you in advance.

Sign in to comment.

Accepted Answer

arushi
arushi on 18 Jan 2024
Hi Bianca,
As per my understanding, it seems that you want to define a transition matrix for an MDP with 50 states and 5 actions, where certain transitions are possible based on the current state and the action taken. The code does not work if you do not consider the other transitions. In some MDP implementations, especially when using certain toolboxes or libraries, it's necessary to define transitions for all state-action pairs, even if some actions are not applicable in certain states. This is typically done to avoid undefined behavior in the MDP solver or to comply with the requirements of the specific MDP framework being used.
To handle this in the code, you can assign a self-transition (transition back to the same state) with a probability of 1 for actions that are not applicable in each state. This effectively means that if you take an infeasible action in a state, you stay in the same state with certainty.
Hope this helps.

More Answers (0)

Categories

Find more on Characters and Strings in Help Center and File Exchange

Products


Release

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!