Training with "trainscg" does not update the networks weights

I am trying to train a very complex though small NARX-Net (see picture).

The weights of the layers 4,5,6,7, as well as layer 2 have been set constant (by disabeling the .learn property of the weights). The transferfunction of the second layer is a sine. All of the layers have been unit tested independently. The backpropagation ect. for the custom sine function works fine.

My problem is, when trying to train the complete structure with train(net) and the trainscg trainFcn, the nntool "trains" the net, but doesn't update the weights. There is a relatively large gradient and also a performance is computed, but as the nntraintool runs through the epochs, the performance and the gradient stays constant. After stopping the training and checking the weights, the are the same as before the "training".

Has anybody experienced similar problems or has a idea what might have gone wrong?

Appreciate your Help,

Christoph

Answer 1

[Rohit]

Hi Christoph,

I understand that there is large gradient and "nntool" is not able to update the weights in your case.

There could be several reasons why the weights of your network are not being updated during training with the "trainscg" algorithm. Here are a few things you can check:

1. Learning rate: Make sure the learning rate is set appropriately. If the learning rate is too small, the weight updates may be too small to make a noticeable difference. Conversely, if the learning rate is too large, the weight updates may overshoot the optimal values and prevent convergence. Experiment with different learning rates to find an appropriate value.
2. Input scaling: Check if the inputs to your network are properly scaled. If the input values are too large or too small, it can affect the training process and make it difficult for the algorithm to converge. Consider normalizing or scaling your input data to a suitable range.
3. Network architecture: Verify that your network architecture is appropriate for the problem you are trying to solve. If the network is too complex or too small, it may struggle to learn the underlying patterns in the data. Experiment with different network architectures, such as increasing the number of hidden layers or neurons, to see if it improves the training performance.
4. Initialization: Ensure that the initial weights of the network are set randomly. If all the weights are initialized to the same value, it can prevent the network from exploring different weight configurations during training. Randomly initializing the weights helps in breaking the symmetry and allows the network to learn more effectively.

By examining these aspects, you should be able to identify potential issues that might be causing the weights to remain unchanged during training.