Clear Filters
Clear Filters

validation error in neural network

3 views (last 30 days)
hossein
hossein on 8 Feb 2015
Edited: Greg Heath on 11 Feb 2015
Dear friends I have tried to train a mlp using newff and train functions, but after few stages of training, the validation error stops the training procedure, so I wanna ask for any solution or alternative to prevent the validation error
regards

Accepted Answer

Greg Heath
Greg Heath on 11 Feb 2015
Edited: Greg Heath on 11 Feb 2015
% 0. One hidden layer is sufficient provided there are enough hidden nodes.
% 1. Use t for target and y for output
% 2. For regression use the current FITNET instead of the obsolete NEWFF.
% 3. As indicated below, Validation Stopping is useful for preventing highly biased training data performance from influencing the design of a net which is ultimately created for use on NONDESIGN data. This is especially important when the number of training equations Ntrneq = Ntrn*O is not sufficiently greater than the number of unknown weights Nw = (I+1)*H+(H+1)*O.
% 4. Data divisions
DATA = DESIGN + NONDESIGN
DESIGN = TRAINING + VALIDATION
NONDESIGN = TEST + UNAVAILABLE
NONTRAINING = VALIDATION + NONDESIGN
% 5. Use the DESIGN TRAINING data to estimate weights. Using the same data to estimate performance can result in highly optimistic biased estimates.
% 6. Use the DESIGN VALIDATION data to obtain less biased performance estimates to
a. Prevent worse performance on NONDESIGN data.
b. Rank the performances of multiple designs
c. Choose the multiple designs used to estimate summary performance statistics
% 7. Assume the UNBIASED NONDESIGN TEST data performance is representative of the performance on UNAVAILABLE data.
% 8. Evaluate performance via the summary statistics of the UNBIASED performance on NONDESIGN (TEST + UNAVAILABLE) data chosen in 6c.
close all, clear all, clc, plt=0
x = -5:.05:5;
t = (2*sin(x).*cos(3*x)+cos(10*x)).*sin(x);
[ I N ] = size(x) % [ 1 201]
[ O N ] = size(t) % [ 1 201]
Ntst = round(0.15*N) % 30
Nval = Ntst % 30
Ntrn = N-Nval-Ntst % 141
plt = plt+1, figure(plt)
plot( x, t, 'LineWidth', 2 )
%17 local minima / 18 local maxima => 36 hidden nodes
H = 36
net = newff( x, t, H );
MSE00 = var(t',1) % 1.0041 Reference MSE
rng(4151941) % For duplicating the design
[ net tr y e ] = train (net, x, t );
hold on
plot( x, y, 'r', 'LineWidth', 2 )
NMSE = mse(e)/MSE00 % 2.5414e-3
R2 = 1-NMSE % 0.99746 Rsquared (See Wikipedia)
tr = tr % No semicolon
stopcrit = tr.stop % Validation stop
R2trn = 1-tr.best_perf/MSE00 % 0.99889
R2val = 1-tr.best_vperf/MSE000 % 0.99542
R2tst = 1-tr.best_tperf/MSE00 % 0.99277
% Using MSEtrn00 instead of MSE00 shouldn't make much difference (Use tr.trainInd to check if you don't believe me)
Hope this helps.
Thank you for formally accepting my answer
Greg

More Answers (1)

Greg Heath
Greg Heath on 9 Feb 2015
The validation stopping occurs because the net is performing badly on nontraining design data. You don't want to overcome it; You want to start designing a better net which works well on BOTH design training data (train) AND design nontraining data (validation).
I typically use a double for-loop to train at least 10 different nets for each of ~Ntrials = 10 candidate values of H = Hmin:dH:Hmax (<=Hub), the number of hidden nodes. The nets differ by the initial state of the random number generator which determines both initial weights AND the data division.
I have posted many examples in the NEWSGROUP and ANSWERS. Search on subsets of
greg, Hmax or Hub, Ntrials, fitnet or patternnet
Hope this helps,
Thank you for formally accepting my answer
Greg
  1 Comment
hossein
hossein on 10 Feb 2015
Edited: hossein on 10 Feb 2015
Dear Greg
Thank you for your answer, as far as I understand from your answer, you mean that I should try with different kind of NN by differing the number of hidden layers and nodes, but it didnt work correctly.
Here I prepared a simple code for this problem, and each time I differed the matrix XX1 for differing the number of nudes and number of hidden layer, but each time the validation error stops the training :
x=-5:.05:5;
y=(2*sin(x).*cos(3*x)+cos(10*x)).*sin(x);
net1=newff(x,y,[XX1]);
net1 = train(net1,x,y);
o1=sim(net1,x);

Sign in to comment.

Categories

Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!