For loop in custom loss function of deep neural network

Hi,

Can anyone tell me whether matlab allows for loop in the customized loss function of a deep neural network?

Does it support complex number calculation in the custom loss function as well?

Answer 1

[Mahesh Taparia]

Hi Jubeyer

You can define a custom loss function in MATLAB, for example you can refer to this link. You can use for loop inside this loss function. Using for loop in the loss function will slower the training process as this function will be called for each batch of each epoch.

[Jubeyer Rahman]

How about the complex number calculation (multiplication) in custom loss function?

[Mahesh Taparia]

From where you will get complex number and what you will do with that?

You can do any computation in loss function but need to take care in backpropagation.

[Jubeyer Rahman]

That complex number will come from outside loss function basically it's a matrix of complex number that will get multiplied by the input and target values.

Since, I have gone through the documentation of custom loss function, it says that the backward loss is optional , for my case the loss function is very complicated since it involves some for loop to produce the mismatch between the input and target. Do I still need to define my backward loss by taking derivative at each step while I assumed that MATLAB will take care of it automatically.

[Mahesh Taparia]

Hi

You can calculate the mismatch simply taking the difference of matrices and you can sum them to compute the net mismatch loss.

If you are doing backpropagation with that loss function, you can use dlgradient to compute the automatic differentiation which will give gradient of the loss function and you can update the weights. For example you can consider the backpropagation using dlgradient here.

[Jubeyer Rahman]

I have gone through the dlarray documentation, but doesn't seem like executable in my case, you can see from my code that a for loop is unavoidable here.

Let me put my code to better explain what I want;

%Y, T (each sample of Y and T is 1X1000)

%B is a matrix (500X500 of complex number)

function loss=forwardloss(layer,Y,T)

global B;

for i=1:size(T(:,1))

V_Y=(Y(i,1:500))'.*exp(sqrt(-1)*(Y(i,501:1000))');

V_T=(T(i,1:500))'.*exp(sqrt(-1)*(T(i,501:1000))');

Apparent_Y(:,i)=V_Y.*conj(B * V_Y);

Apparent_T(:,i)=V_T.*conj(B * V_T);

end

loss=sum(Apparent_T-Apparent_Y);

end

[Jubeyer Rahman]

The loop will iterate upto the number of samples in each batch. In the code, it is wrong in the for loop statement starting line.

[Mahesh Taparia]

How about writing it as

Q=size(T(:,1));

V_Y=(Y(1:Q,1:500)).*exp(sqrt(-1)*(Y(1:Q,501:1000)));
V_T=(T(1:Q,1:500)).*exp(sqrt(-1)*(T(1:Q,501:1000)));
Apparent_Y=V_Y.*conj(B * V_Y);
Apparent_T=V_T.*conj(B * V_T);
loss=sum(sum(Apparent_T-Apparent_Y));

I think it should work. Use dlgradient to calculate the gradient for backpropagation.

[Jubeyer Rahman]

Thank you very much. Looks like it may work, I will get back after testing this.

[Jubeyer Rahman]

To make things simple let's say Q is 20

looks like the inner multiplication in line 4, where conj (B*V_Y) will fail it since B is 500X500 whereas each row of V_Y needs to be multiplied with the whole matrix B, here the inner dimension will not match.

[Mahesh Taparia]

Take the transpose of V_Y as you already did in your code previously.

[Jubeyer Rahman]

I think I should be more specific about the problem I am facing here:

Here, in this operation:

Apparent_Y=V_Y.*conj(B * V_Y);

Let's say V_Y is 500X20 and B is 500X500

From vectorized operation what I need is when B*V_Y will be executed: it has to be in a way that each individual column of V_Y has to be multiplied with the whole B matrix and the result will be another 500X20 matrix (each time a column of V_Y is multiplied with the whole B, a column of 500X1 will be created; these columns need to be in a concatenated form which will make the desired resultant matrix of 500X20 matrix). So, just transposing V_Y does not solve the problem neither the elementwise multiplication, and I am doubtful whether while taking derivative any repmat type command is allowed or not.