LSTM Custom Regression output layer for time series

Hello everyone!

I'm implementing an LSTM with custom regression output layer, as shown in https://www.mathworks.com/help/deeplearning/ug/define-custom-regression-output-layer.html.

Ideally, I would like to have as extra input variable of the forward and backward fuctions the input of the network, but in the toolbox the function is limited to layer, Y and T.

My structure is made by:

• layer1 = sequenceInputLayer(numFeatures);
• layer2 = lstmLayer(numHiddenUnits);
• layer3 = fullyConnectedLayer(numResponses);
• layer4 = myRegressionLayer;

I've created myRegressionLayer (layer, T, Y, X) and I've tried to modify the functions in both in nnet.cnn.layer.RegressionLayer and nnet.cnn.layer.Layer to expect an extra input variable (the network input, X), but I get an error from CNNException.m:

"Error in LSTM (line 79)

[net,temp] = trainNetwork(XTrain,YTrain,layers,options);

Caused by:

Insufficient number of outputs from right hand side of equal sign to satisfy assignment."

Anyone that can help me with this? I need to pass XTrain to my regression layer alongside layer, Y and T.

I'm working with time series, 2 features in input and one output.

Thanks!!

Silvia

Answer 1

[Raunak Gupta]

Hi,

For adding Multiple Inputs or Learnable parameters to a Custom Layer you may try following below mentioned documentations.

Define Custom Deep Learning Layer with Multiple Inputs

Define Custom Deep Learning Layer with Learnable Parameters

Note that for Custom Layers you may have to define forward and backward function which can be non-intuitive to derive from the mathematical form especially the derivative of loss function is case of backward function.

[Silvia Cisotto]

Hi Raunak,

thank you very much for your help!

How can I access the code of the forward and backward functions currently implemented in the toolbox regressionLayer? I have tried help and open but it doesn't work. I would like to see how the two functions are derived.

Thanks!

Silvia

[Raunak Gupta]

Hi Silvia,

For every custom layer forward function function means waht the output of the layer will be and backward function is required for backpropagating the loss to previous layer. Forward function is easy to write but backward function can be difficult because it requires deriving the derivative of the Loss function.These function are what makes the layer "custom" So, you may have to write them on your own in code. In the documentation mentioned above some examples are given on the structure of custom layers and the functions inside.

Note that it is required to create a new file with name as Layer name for custom layers.

[Silvia Cisotto]

Hi Raunak,

thank you for your reply. Just to better clarify with a practical example, in here, the architecture is the same I'm using, with an input layer, a LSTM layer, a fully connected layer and a regression layer.

If I got it right, the forward and backward funtions in this case should be implemented in the regression output layer, as the structure of this layer should be as shown in the example of the custom output regression layer here.

I'm trying to find and open these functions, to see how in the "Time Series Forecasting using LSTM" example the backward function has been derived from the Loss function, but I cannot find them.

How/Where can I access and see the derivation of the Loss function for the backpropagation function implemented in the regressionLayer in the Time Series Forecasting Using Deep Learning for this specific case?

Thank you again for your help.

Silvia

[Raunak Gupta]

Hi,

The backward loss is not mentioned in here since the loss function implemented in forward has been using only dlarray and for which defining backward function is optional.

Since in this model multiple inputs are present it is required to define backward as it depends on all inputs. You can refer to following documentation for such examples. The main essence is to use function defined especially for writing custom training loops and layers. dlgradient will help writing the derivative of loss function.

Automatic Differentiation , dlgradient , dlfeval

Custom Training Loops

[Silvia Cisotto]

Hi Raunak,

thanks for your suggestions.

I have another question regarding the topic above. How can I retrive the training dataset I feed to the sequenceInputLayer inside the regression output layer? I'm sure that there is a command line for this but I cannot find it.

Thanks!!

Silvia

[Raunak Gupta]

Hi,

Since you want to retrieve the output from regression output layer with inputs coming from sequenceInputLayer I suggest connecting the two layers first with connectLayers so that the output from sequenceInputLayer can be extracted inside the regression output layer. This way you can get the training dataset as a input parameter in custom regression output layer. Make sure to change the code accordingly to get extra input data as both layers will be connected.

[Silvia Cisotto]

Hi,

thanks for the feedback. Quick question, given that I need to add an extra input to connect the input layer to the regression layer, I guess that I cannot use the 'custom regression output layer' layout/example and modify this and therefore I need to create my own regression output layer using the multiple input custom layer layout/example. Is this correct? If so, do I then need to define the derivate with respect to both the fully connected layer and the input layer?

Also, I noticed that in CustomLayer, line88, there is a line of code 'this.NeedsXForBackward = iIsXUsedInBackward( anExternalCustomLayer ); ' . Is the X mentioned here, the input data or is this something else?

Thanks!

Silvia

[Raunak Gupta]

Hi,

For the first question, yes you may need to define the custom layer which can take multiple inputs but by mentioning learnable parameters in the layer to be only the weights of that custom regression layer you may retrieve the data without creating any other learnable parameter. Therefore the derivative should only contain parameters of the fully connected layer.

For the second question, X is the data that is feed to the layer as input rather than the original training data.You may take training data via connectLayer into one more variable X2.

Hope everything is clear now. You may take a closer look to the two documentation that I mentioned in the Answer.

[Shubham Baisthakur]

Hello Raunak and Silvia,

I am facing exactly the same issue and I was wondering if you wiil be able to help. Follwing is the definition of my custom regression layer

classdef customLossLayerMultiInput < nnet.layer.RegressionLayer & nnet.layer.Acceleratable
    % Custom regression layer with mean-absolute-error loss and additional properties.
    
    properties
        node_properties
        numFeature
    end
    
    methods
        function layer = customLossLayerMultiInput(name, node_properties, numFeature)
            % Constructor
%             layer.NumInputs = numInputs;
            layer.Name = name;
            layer.Description = 'Physics-Informed loss function for LSTM training';
            layer.node_properties = node_properties;
            layer.numFeature = numFeature;
        end
        
        function loss = forwardLoss(layer, Y, T, varargin)
            % Calculate the forward loss
            
            % Reshape predictions and targets
            Y = reshape(Y, [], 1);
            T = reshape(T, [], 1);

            X1 = varargin{1};
            X2 = varargin{2}; 

            % Sequence input data
            sequence_input_data = reshape(X1, [], layer.numFeature);

            % Calculate mean residue
            mean_residue = PI_BEM_Residue(Y, T, sequence_input_data, layer.node_properties);

            % Calculate RMSE loss
            rmse_loss = rmse(Y, T);

            % Total loss
            loss = mean_residue + rmse_loss;
        end
    end
end

And my network architecture is:

layers = [
    sequenceInputLayer(numFeatures, 'Name', 'inputLayer') % Define the sequence input layer and name it
    lstmLayer(num_hidden_units, 'OutputMode', 'sequence', 'Name', 'lstmLayer') % Define the LSTM layer and name it
    fullyConnectedLayer(1, 'Name', 'fullyConnectedLayer') % Define the fully connected layer and name it
    dropoutLayer(x.dropout_rate, 'Name', 'dropoutLayer') % Define the dropout layer and name it
    customLossLayerMultiInput(LayerName, node_properties,numFeatures)
    ];

% Create a layer graph
lgraph = layerGraph(layers);
lgraph = connectLayers(lgraph,"inputLayer",strcat(LayerName,'\in2'));

However I am not able to have two inputs for the custom loss layer. Now as I understand, the custom regression layer can not have multiple inputs. But if I want to use a "nnet.layer.Layer" class to define a custom layer with mutiple inputs, how do I access Y and T in that layer? Also how do I set this layer to work as a regression layer?