Multi Input Multi Output Machine Learning model for varying length sequence data.

25 views (last 30 days)
Dear reader,
I have been trying to create a Multi Input Multi Output Machine Learning model in Matlab for sequence data which is varying in length. The source of inspiration for my solution has been the following two examples:
After not getting it to work in my main code, I created (together with chatgpt) this example code to test certain variations.
close all, clear all, clc
% Step 1: Prepare sequence data (input and output) with varying lengths
% Data in [Features x Time] format
input1 = {rand(3,10)'; rand(3,8)'; rand(3,12)'}; % Input 1: 3 features
input2 = {rand(5,10)'; rand(5,8)'; rand(5,12)'}; % Input 2: 5 features
output1 = {rand(2,10)'; rand(2,8)'; rand(2,12)'}; % Output 1: 2 features
output2 = {rand(1,10)'; rand(1,8)'; rand(1,12)'}; % Output 2: 1 feature
% Combine input data into cell array
inputs = {input1, input2};
outputs = {output1, output2};
% Step 2: Define Network Architecture with multiple inputs and multiple outputs
% Input layers for each input
inputLayer1 = sequenceInputLayer(3,'Name','input_1'); % 3 features for input 1
inputLayer2 = sequenceInputLayer(5,'Name','input_2'); % 5 features for input 2
% LSTM layers for input processing
lstmLayer1 = lstmLayer(10,'Name','lstm_1'); % Processes input 1
lstmLayer2 = lstmLayer(20,'Name','lstm_2'); % Processes input 2
% Concatenation layers (concatenating along channels dimension)
concatLayer1 = concatenationLayer(1,2,'Name','concatLayer1');
concatLayer2 = concatenationLayer(1,2,'Name','concatLayer2');
% Fully connected layers for the outputs
fc1 = fullyConnectedLayer(2,'Name','fc_1'); % Output 1: 2 features
fc2 = fullyConnectedLayer(1,'Name','fc_2'); % Output 2: 1 feature
% Create layer graph with branching for multiple inputs and outputs
layers = layerGraph();
% Add and connect layers for input 1
layers = addLayers(layers, inputLayer1);
layers = addLayers(layers, lstmLayer1);
layers = addLayers(layers, concatLayer1);
layers = addLayers(layers, fc1);
% Add and connect layers for input 2
layers = addLayers(layers, inputLayer2);
layers = addLayers(layers, lstmLayer2);
layers = addLayers(layers, concatLayer2);
layers = addLayers(layers, fc2);
% Connect network branches
layers = connectLayers(layers, 'input_1', 'lstm_1');
layers = connectLayers(layers, 'input_2', 'lstm_2');
layers = connectLayers(layers, 'lstm_1', 'concatLayer1/in1');
layers = connectLayers(layers, 'lstm_2', 'concatLayer1/in2');
layers = connectLayers(layers, 'lstm_2', 'concatLayer2/in1');
layers = connectLayers(layers, 'lstm_1', 'concatLayer2/in2');
layers = connectLayers(layers, 'concatLayer1', 'fc_1');
layers = connectLayers(layers, 'concatLayer2', 'fc_2');
% Convert to dlnetwork object
dlnet = dlnetwork(layers);
% Step 3: Define custom loss function for use with trainnet
function loss = modelLoss(dlnet, dlX, dlY)
% Extract individual inputs and outputs
dlX1 = dlX{1};
dlX2 = dlX{2};
dlY1 = dlY{1};
dlY2 = dlY{2};
% Forward pass through the network
dlYPred1 = forward(dlnet, dlX1, 'input_1');
dlYPred2 = forward(dlnet, dlX2, 'input_2');
% Compute loss for both outputs
loss1 = mse(dlYPred1, dlY1);
loss2 = mse(dlYPred2, dlY2);
loss = loss1 + loss2;
end
% Step 4: Create a datastore for training
inputDatastore = arrayDatastore(inputs, 'IterationDimension', 1);
outputDatastore = arrayDatastore(outputs, 'IterationDimension', 1);
% Combine the input and output datastores
ds = combine(inputDatastore, outputDatastore);
% Training options
options = trainingOptions('adam', ...
'MaxEpochs', 100, ...
'MiniBatchSize', 3, ...
'Plots', 'training-progress');
% Display the network structure
figure
plot(layers);
% Step 5: Train the network using trainnet
trainedNet = trainnet(ds, dlnet, @modelLoss, options);
Sadly, the following error occurs:
Error using trainnet (line 46)
Error forming mini-batch for network input "input_1". Data interpreted with format "TCB". To specify a different format, use the InputDataFormats option.
Error in test_chatgpt (line 97)
trainedNet = trainnet(ds, dlnet, @modelLoss, options);
Caused by:
Input sequences must be numeric or categorical arrays.
The error says that the data is interpreted as TCB, the sequence example I think also uses TCB after testing several options (default is 'auto'). Therefor the error might not be the actual error.
So I'm wondering, are there options to solve this? As far as I understand does the data handler not understand how to read the data eventhough both examples show the possibility for it individually.
One solution would be creating a 3D array with padded data. But I would not prefer to use padded data since that could have effect on the model since different lengths resemble different speeds in my scenario and that could influence the model.
Another solution would be to make all the samples the same length without padding and adding them to a 3D array. However that would reduce the amount of data for the model to train, but does not introduce an indirect variable to the model.
Thanks in advance for your time.

Answers (1)

Gayathri
Gayathri on 10 Oct 2024
I understand that you are facing error while trying to train a “Multiple Input Multiple Output” Neural Network.
To solve the error “Input sequences must be numeric or categorical arrays.”, we can first convert the inputs and outputs from cell array to numeric arrays before converting it to an “arrayDatastore” object. And as the network expects two inputs and two outputs, it is advised to combine the “arrayDatastore” objects separately for each input and output. Please refer to the below codes for your reference.
convertToNumeric = @(cellArray) cellfun(@(x) double(x), cellArray, 'UniformOutput', false);
% Convert cell arrays to numeric arrays
inputnumeric1 = cell2mat(convertToNumeric(input1));
inputnumeric2 = cell2mat(convertToNumeric(input2));
outputnumeric1 = cell2mat(convertToNumeric(output1));
outputnumeric2 = cell2mat(convertToNumeric(output2));
% Create datastores for each input and output
inputDatastore1 = arrayDatastore(inputnumeric1, 'IterationDimension', 1);
inputDatastore2 = arrayDatastore(inputnumeric2, 'IterationDimension', 1);
outputDatastore1 = arrayDatastore(outputnumeric1, 'IterationDimension', 1);
outputDatastore2 = arrayDatastore(outputnumeric2, 'IterationDimension', 1);
% Combine the datastores into a single datastore
ds = combine(inputDatastore1, inputDatastore2, outputDatastore1, outputDatastore2);
Next, we can make one additional modification to the code provided. I understand that, you want to use “mse” as loss function for both the set of data. Therefore, please mention the loss function in the below format and then pass it to the “trainnet” function.
lossFcn = @(Y1,Y2,T1,T2) mse(Y1,T1) + mse(Y2,T2);
% Step 5: Train the network using trainnet
trainedNet = trainnet(ds, dlnet, lossFcn, options);
For more information on training a “Multiple-Input and Multiple-Output Networks”, please refer to the below mentioned link.
Hope you find this information helpful.
  1 Comment
Corné Dijkstra
Corné Dijkstra on 10 Oct 2024
Dear Gayathri,
Thanks for your quick reply. Your solution does make the ML run, however the purpose of the ML would be to learn from different subjects and combining the data into 1 set without separation would not be the desired outcome.
Maybe a general question to Mathworks, how are datastores supposed to function? I compare them to pointes in C++, pointing to the data to be used. Since in the sequence example a cell array does function for trainnet, I cannot fathom why a cell array would not function in combination with a datastore. Evenmore, why does for a MIMO system the input and output be placed together in the datastore? It makes it only more difficult for the code to determine what is supposed to be what. Since apart from size I haven't found another determining factor how it differeniates which data should be connected where in the ML.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!