Good value for root mean square error?

95 views (last 30 days)
imo88
imo88 on 1 Feb 2017
Commented: Jonathan on 19 Feb 2019
Hello,
I was working on the calculation of some RMSE values. I have some force trials in the time domain (normal force, and tangential force). I wanted to see if the reproduction of these forces was accurate when a robotic platform reproduces the same force we apply. I calculated the RMSE for all the trials I have (450).
The mean RMSE for the normal force was 0.11 (SD=0.08) and for the tangential force was 0.15 (SD=0.1). Do you think that these are decent values to conclude that the reproduction of the force was quite accurate?
Thank you very much in advance for your help!
ps: I also computed the root mean square range error that was found to be even lower. Does it also make sense to compute the range error?
  4 Comments
Jonathan
Jonathan on 19 Feb 2019
rms function was about 15 times slower than a simple manual computation! My simple manual computation:
rmse = sqrt( (length(Y)^-1).*sum((Y_hat - Y).^2) )
where Y is your target data, and Y_hat is your estimate.

Sign in to comment.

Accepted Answer

John D'Errico
John D'Errico on 23 Feb 2017
As dpb said, it is impossible to know if some arbitrary value for RMSE is good or bad. Only you know if it is good, because only you can know how much noise you would expect in the data.
The point is, when you use a model on some data that generates an RMSE, there are TWO components to the error, noise and lack of fit.
Any model will not be able to predict random noise in the data, so the predictive capability of the model can be no better than that noise. If it is better, then you are overfitting the noise. This is a bad thing to do.
At the same time, a model is just a model. It is an approximation to the real thing, with some aspects of reality that are left out. In some cases, a model is just a metaphor for the real system under study, so your model arises from some completely different system, but one that you hope behaves similarly to the real world case under study. In either case, a model will typically have some degree of lack of fit, since it is never a perfect model of the process. There is always something you have not included.
So we think of the residuals for any model to a physical system as something composed of both noise and lack of fit. These two components mix together. If the model is a good one, and the lack of fit is very small, then all we will see after the model is removed is noise. So our hope is the RMSE will be roughly the same as the underlying noise standard deviation, something that you SHOULD have some feel for if you are modeling any process.
So if the RMSE is significantly very small, more so than we would expect, then you are overfitting the problem. That is bad, since your model is now compensating for (chasing after) random noise.
If the RMSE is significantly larger than you would have expected, then this is again a problem, since it indicates there is a potential problem in the model. The model is inadequate to represent the system under study, or you did a poor job of estimating anything that must be estimated in the model.
But the point is, RMSE means NOTHING unless you have some a-priori knowledge about the process. You must have some expectation as to what is acceptable for the problem you are studying. IF you have no clue, it is impossible for someone else to know, since did not develop the model, we did not generate the data under study.

More Answers (0)

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!