How could I remove outliers in really small data?

3 views (last 30 days)
I have data that should resemble a parabola when plotted into a figure. However, near the center, there is a "high" value for the data.
x = [-9.0000 -8.0000 -7.0000 -6.0000 -5.0000 -4.0000 -3.0000 -2.0000 -1.0000 0 1.0000 ...
2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 10.0000];
y = [0.0173 0.0169 0.0168 0.0166 0.0166 0.0167 0.0165 0.0165 0.0166 0.0167 0.0168 ...
0.0177 0.0189 0.0173 0.0176 0.0178 0.0180 0.0181 0.0182 0.0185];
The values I would like Matlab to see as an outlier are x = 2 --> y = 0.0177, and x = 3, --> y = 0.0189, because I should not expected the parabola to grow in the middle, and then decrease. However, it does not count this points as outliers because, of course, Matlab does not know that I should be expecting a parabola-like shape. How could I do this? Thank you!
  6 Comments
John D'Errico
John D'Errico on 20 Nov 2021
What DPB has suggested is a variation of often called an iteratively reweighted least squares. It is the basis for many of the robust fitting tools you will find. Points with large residuals are downweighted, then a weighted fit is redone. In the case of an outlier scheme, you can just decide to just remove them if you wish.
Juan Manuel Hussein Belda
Juan Manuel Hussein Belda on 22 Nov 2021
Thank you both! By the way, John, your function inpaint_nans is a life saver!!! (Do not worry, I have properly cited it everytime I used it :) )

Sign in to comment.

Accepted Answer

Sulaymon Eshkabilov
Sulaymon Eshkabilov on 21 Nov 2021
Linear interpolation might be good to use here, e.g.:
x = [-9.0000 -8.0000 -7.0000 -6.0000 -5.0000 -4.0000 -3.0000 -2.0000 -1.0000 0 1.0000 ...
2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 10.0000];
y = [0.0173 0.0169 0.0168 0.0166 0.0166 0.0167 0.0165 0.0165 0.0166 0.0167 0.0168 ...
0.0177 0.0189 0.0173 0.0176 0.0178 0.0180 0.0181 0.0182 0.0185];
plot(x,y, 'linewidth', 2), shg
% Linear Interpolation
x1 = 2; y1 = 0.0177;
x2 = 3; y2 = 0.0189;
Idx = find(x==x1 | x==x2);
y(Idx) = interp1([x(Idx(1)-1),x(Idx(2)+1)], [y(Idx(1)-1),y(Idx(2)+1)], x(Idx));
hold on
plot(x, y, 'r--', 'linewidth', 2), grid on; legend('Raw: x vs. y', 'Fixed: x vs. y')

More Answers (0)

Categories

Find more on Programming in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!