How is RMSE calculated on Multivariate Regression Neural Networks?

In short: The multivariate regression network ends with validation RMSE of e.g. ~72.6 while my own calculation ends up to be 14.8. How is this network calculating the RMSE on multivariate regression? How can I trust this RMSE / Loss optimization? Did I miss some configuration?

At first I thought those 72.8 is the RMSE for a whole day and 72.8/24 = 3.0 will be my RMSE for a single hour. But that does not fit on my own RMSE calculation afterwards, which states 14.8 hourly RMSE. I use the following formula:

RMSE = sqrt(mean((v1-v2).^2));

Before my RMSE calculation, I reshape the prediction/validation data to Kx1 matrix, where K represents the hours of each day in order.

Some facts on my approach:

• My feature array has the shape of NxM, where N is the number of features and M is the corresponding day (in order).
• My label set look similar: 24xM, where 24 represents the hourly energy price and M is the corresponding day (in order).
• I split the data beforehand into training & validation data (while cut out some random days as testing data beforehand).

As far as I can tell, that seems to work with 24 responses as regression output. But the RMSE is far away from being reasonable.

Any help will be appreciated! :)

Network:

numFeatures = size(features.input_data, 1);
numHiddenUnits1 = 250;
numResponses = 24;

layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits1,'OutputMode','sequence')
    fullyConnectedLayer(100)
    fullyConnectedLayer(50)
    fullyConnectedLayer(numResponses)
    regressionLayer];

options = trainingOptions('sgdm', ...
    'MaxEpochs',200, ...
    'ValidationData',{features.val, labels.val},...
    'ValidationFrequency',5,...
    'InitialLearnRate', 0.05,...
    'LearnRateSchedule', 'piecewise', ...
    'LearnRateDropFactor',0.8, ...
    'LearnRateDropPeriod',15, ...
    'Plots',"training-progress", ...
    'Verbose', false, ...
    'GradientThreshold', 0.25);

[net scores] = trainNetwork(features.train, labels.train, layers, options);

Answer 1

[Prasanth]

I am assuming that you are using MATLAB R2020a and you tried to design a neural network with a 'regressionLayer' as output. You computed the validation error from your neural network as ~72.6 (example).

By default, the 'regressionLayer' calculates the half mean squared error. So, when you are referring to the validation loss given by your neural network, it is actually the half mean squared error, but not RMSE, as you are assuming.

You can refer to the documentation link for the regressionLayer for more details.

Currently, there is no option to choose the loss function in the 'regressionLayer'.

If you want to use any other loss function, you can create a custom regression layer with your own loss function.

You can refer to Defne Custom Regression Layer documentation link to get more information and template code for creating a custom regression layer. The example in the above link creates a custom regression layer with Mean Absolute Error (MAE) as the loss function.

I hope this helps in resolving your issue.

[Raphael Rottshäfer]

Hello Prasanth,

thank you very much for your detailed explanation on my issue! You are totally right: I use R2020a and a regressionLayer. And now I know what is happening there. As I assume, that the vanilla regression layer's loss function scales as expected, I skip the Cutom Regression Layer for now, since now I can comprehend the vanilla version. :)

Thank you!