In answers like this, Greg Heath suggests using the normalized mean square error, NMSE, to compare the performance of different neural networks and pick the best one.
I have been calculating NMSE from all samples in the training set t and prediction y,
[net tr y e ] = train(net,x,t);
vart1 = var(t',1);
MSE00 = mean(vart1);
NMSE = mse(t-y)/MSE00;
That includes the training samples, and so may favor models that fit the training data well but not new data. In order to choose the most robust model, should I calculate NMSE from the test samples only?
iTest = tr.testInd;
NMSE_test_only = mse(t(:,iTest)-y(:,iTest))/MSE00;