Problem with matrix function output (Q learning, Calvano et al. 2021)

Hello !

I'm currently working on the Q learning algorithm similar to the one presented in Calvano et al. 2021 (AER).

As a part of preliminary work, I want to produce a distribution table similar to the one I attached below.

The idea is simple. I have a function which gives prices and profits for 2 market makers depending on multiple parameters. More particularly, I want it to vary in alpha (2.0:0.1:4.0) and beta (0.04:0.001:0.06) and fix other parameters at the begining. Then, having a matrix of average values (my function 'fn' gives average prices and profits) and using imagesc(), it would be easy to obtain the desired results. The problem is that the code is not working.

Here is what I have at the moment :

beta0=0
n=10000
T=1000
[P,Profit]=fn(2.0:0.1:4.0,0.04:0.001:0.06,beta0,n,T); % I don't know how to index the profits depending on alpha and beta in order to have a clear matrix later. 
Matr=append(Profit(:,1)); % Here I put Profit(:,1) because I'm verifying first the profits of the first market maker. But the 'append' function is clearly a wrong idea.
a=(2.0:0.1:4.0)
b=(0.04:0.001:0.06)
imagesc(b, a, Matr);
set(gca, 'YDir', 'normal'); 
xlabel('beta');
ylabel('alpha');
colorbar

I've already spent hours looking for an answer. It's the first time I use matlab which makes it difficult not only to optimize my code (to make it less time consuming) but also to make it work. I would be very greatful for any help. Thank you !!

[Walter Roberson]

Are you thinking in terms of selecting the first alpha and for it, using all of the beta values, and appending the results to a matrix; and then going on to the second alpha and for that using all of the beta values, and appending the results to a matrix, and so on, ending up with a matrix that is length(alpha) by length(beta) ?

If so then possibly your fn() is already producing that kind of output. We would need to see the code for it to be sure.

Your 2.0:0.1:4.0 and 0.04:0.001:0.06 each happen to have 21 elements, so depending on how your fn is working, it is also possible that instead of producing a 21 x 21 output, that it would produce a 1 x 21 output... depends how it was written.

[Olena Bogdan]

Yes, It's exactly what I want to do.

Here is the way I defined my fn.

function [P0,Profit0]=fn(alpha,beta,beta0,n,T)
%Runs n experiments for T periods with the same alpha, beta, and beta0, and computes average
%prices and profits
P0=zeros(T,2); 
Profit0 = zeros(T,2);

for i=1:n
    [P,Profit]=f1(alpha,beta,beta0,T);
    P0 = P0 + P;
    Profit0 = Profit0 + Profit;
end

P0=mean(P0);
Profit0=mean(Profit0);

And my fn1 takes the following expression :

function [P,Profit]=f1(alpha,beta,beta0,T)
%Runs a learning experiment once, for T periods.

%Outcomes
P=zeros(T,2); %Price chosen by each player in each period
Profit = zeros(T,2); %Profit achieved by each player in each period

%Payoff Vectors for Players 1 and 2 - Initialization
%Player 1 has column 1 and Player 2 has column 2
M = zeros(11,2);


t=1:T;
epsilon = max(beta0,exp(-beta*(t)));
%We first compute the price chosen by each player.
s =zeros(1,2); %Will denote the index of the price chosen by each player
for i=1:2
%Random Draw. If X=1 the player explores at this round, otherwise plays argmax M.    
X = binornd(1,epsilon); 

Maybe I need to modify my original functions to have 21*21 output instead of simple vector of profit and price?

[Walter Roberson]

Where does your code use alpha ?

Also your f1 function appears to be truncated.

[Olena Bogdan]

Oh sorry, here the whole code.

function [P,Profit]=f1(alpha,beta,beta0,T)
%Runs a learning experiment once, for T periods.

%Outcomes
P=zeros(T,2); %Price chosen by each player in each period
Profit = zeros(T,2); %Profit achieved by each player in each period

%Payoff Vectors for Players 1 and 2 - Initialization
%Player 1 has column 1 and Player 2 has column 2
M = zeros(11,2);


for t = 1:T
    epsilon = max(beta0,exp(-beta*(t)));
%We first compute the price chosen by each player.
s =zeros(1,2); %Will denote the index of the price chosen by each player
for i=1:2
%Random Draw. If X=1 the player explores at this round, otherwise plays argmax M.    
X = binornd(1,epsilon);    

if X == 1
s(1,i) = randi([0 10],1,1);
P(t,i) = 0.01*s(1,i); %Generates a random price in 0, 0.01... 0.10
end

if X == 0
m = max(M(:,i)); %Maximum of the payoff vector for player i
maxvector = find(M(:,i)==m); %Indices of all the values corresponding to a maximum
s(1,i) = maxvector(randi([1 length(maxvector)],1,1))-1;
P(t,i) = 0.01*s(1,i); %Selects a random index corresponding to a maximum
end
    
end
%Compute the profits in period t
[Profit( t,1),Profit(t,2)] = Profits(P(t,:));

%Update the payoff vectors
M(s(1,1)+1,1) = alpha*Profit(t,1) + (1-alpha)*M(s(1,1)+1,1);
M(s(1,2)+1,2) = alpha*Profit(t,2) + (1-alpha)*M(s(1,2)+1,2);

end

Answer 1

[Sulaymon Eshkabilov]

There are several inconsistencies in your code:

(1) You are calling the function file called fn. However, your function file is named f1.

(2) You are inputting too many inputs (five in total, viz. 2.0:0.1:4.0,0.04:0.001:0.06,beta0,n,T) in fn. However, you have assigned only four input variables, viz. alpha,beta,beta0,T

(3) Another crucial err is that you have not defined//calculated Profits().

If you post the missing parts of your scripts, additional help can be provided.

[Olena Bogdan]

Thank you !

(3) Here the profits function.

function [pi1,pi2]=Profits(P)

p1 = P(1);
p2 = P(2);

if p1 < p2
   pi1 = p1*(0.1-p1)/0.1;
   pi2=0;
end

if p1 > p2
   pi2 = p2*(0.1-p2)/0.1;
   pi1=0;
end

if p1 == p2
   pi2 = 0.5*(p2*(0.1-p2)/0.1);
   pi1=pi2;
end

(2) As for '2.0:0.1:4.0,0.04:0.001:0.06' in the function input, I just wanted to replace the nested for loop, i.e. but maybe it is not the best way to do it...

for alpha = 2.0:0.1:4.0
    for beta = 0.04:0.001:0.06
        [P, Profit]=fn(alpha, beta, beta0, n, T)
    end
end

(1) I use two different names, fn and f1 to disinguish between the rounds and experiments along the learning path.

[Sulaymon Eshkabilov]

There are still two more parameters are ill-defined or overlooked. They are n and M. "n" is never been used and M has only "0" values.

To append all the values of Profit, CELL array can be employed within the loop.