Implementation of the feedforwardnet Neural Network

I have implemented a very simple neural network to estimate a sine function. The following is the code for generating and training the network:

% Generate Data
dataSize = 1000;
x = linspace(0, 2*pi, dataSize);
y = sin(x);
hold off
plot(x,y)
hold on
% Add noise to Data
yInput = y+randn(1,dataSize)./5;

% No need to seperate training, test and validation data, that occurs automatically in train function.

% Generate Network.  A Very simple two layer model, with two nodes in input layer.
net = feedforwardnet([2]);
% Train Network
net = train(net,x,yInput);

% Show result of trained network
yNN = net(x);
figure
plot(x,yNN, '*')
hold on
plot(x,y, '.')

Now my question, how is this network actually implemented. According to literature, I should be able to recreate the network, copying the weights and biases, with the following function:

function [y] = mynet(net, x_val)
%MYNET A manual implementation of the feedforward network, to demonstrate functionality.
    W1 = net.IW{1};
    b1 = net.b{1};
    W2 = net.LW{2};
    b2 = net.b{2};
    y = purelin(W2*tansig(W1*x_val + b1)+b2);
end

However, the original net(x) function, and mynet(x) produce completely different results. Although the weights and biases are exactly the same, the functions are also directly copied over, you can extract them from the network with:

>> net.layers{1}.transferfcn
ans =

    'tansig'
>> net.layers{2}.transferfcn
ans =

    'purelin'

Can anyone suggest where my implementation of the neural network is wrong. I am really hoping that it is a simple mistake, but I just cant see it at the moment.

Many thanks in advance

[Joshua Ogbebor]

Hi

Can you run the code net.inputWeights{1,1}.weightFcn and see if it is dot product?

[SMEAC]

Hello,

yes, I can confirm it is the dot product.

>> net.inputWeights{1,1}.weightFcn

ans =
    'dotprod'
    

And this is essentially what is already programmed. My network only has one input, so the input value x_val is a scalar. And no dot product is calculated. However, moving onto the second layer, where the different nodes with the different weights are brought together, this is evaluated with the vector vector multiplication W2*tansig... The following example shows this is calculating the same:

>> W2*tansig(W1*x_val + b1)

ans =

    0.5891

>> dot(W2,tansig(W1*x_val + b1))

ans =

    0.5891

Answer 1

[Steven Lord]

How are you handling the preprocessing functions and postprocessing functions?

net = feedforwardnet([2]);
net.inputs{1}.processFcns

ans = 1x2 cell array
    {'removeconstantrows'}    {'mapminmax'}

net.outputs{2}.processFcns

ans = 1x2 cell array
    {'removeconstantrows'}    {'mapminmax'}

See the description of the inputs, outputs, etc. subobjects on this documentation page for more information.

[SMEAC]

Yes!! Many thanks.

I hadn't seen that the processFcns work as pre- post-processing functions.

On my network, the

net.inputs{1}.processFcns and net.outputs{2}.processFcns both give a single cell answer of the mapminmax function.

When I modify my code to include this it works perfectly.

For Info, in my implementation, I have chosen not to use directly the mapminmax function, because it requires that the entire range is given as an input, making it impossible to use the network to evaluate a single output. But as it is a linear mapping function, it is essentially just a further gain and offset, or weight and bias. :)

Answer 2

[Joshua Ogbebor]

In general the code net(x) is equivalent to sim(net, x), and this implementation depends on so many properties of the feedforward nnetwork that you may not be aware of. I copied this from the sim function page and thought t would be helpful. You could investigate and see if any one of these could be the issue.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

sim uses these properties to simulate a network net.

net.numInputs, net.numLayers

net.outputConnect, net.biasConnect

net.inputConnect, net.layerConnect

These properties determine the network’s weight and bias values and the number of delays associated with each weight:

net.IW{i,j}

net.LW{i,j}

net.b{i}

net.inputWeights{i,j}.delays

net.layerWeights{i,j}.delays

These function properties indicate how sim applies weight and bias values to inputs to get each layer’s output:

net.inputWeights{i,j}.weightFcn

net.layerWeights{i,j}.weightFcn

net.layers{i}.netInputFcn

net.layers{i}.transferFcn

[SMEAC]

I have been looking at most of these options, but they dont make much of an influence in the implementation here.

For example, all delays are 0:

>> net.inputWeights{1}.delays
ans =

     0
>> net.layerWeights{2}.delays

ans =

     0

The input function is the dot product what I discussed above and the transfer functions are the tansig and purelin which are implemented.

To use the slides from Prof. Ricardo de Castro from the TU Muenchen, the implemenation of sim(net, x) and net(x) as well as my implementation of mynet(x) should all be exactly the same based on the following description of a basic neural network.

Which in the MATLAB Code is shown by:

Unfortunately, the actual implementation in matlab is hidden in nnMex.yy() a compiled mex function where the code is not visible. So I cant see why I have made a mistake or how they come to their conclusion.