How to specify max. number of outliers on which to apply the Hampel filter?

2 views (last 30 days)
I have a timeseries with outliers. However, sometimes these occur consequtively, e.g. 20 datapoints one after the other. This poses a problem when applying Hampel on the entire dataset because it alters data that is actually valid.
I would like to apply the Hampel filter only when there are max. 3 outliers in a Hampel window. How can I do that?
The outliers are values within the vector X that exceed a threshold of abs(X) = 10.
Is there a way to create a new vector which keeps datapoints exceeding 10 if most of the sorrounding datapoints are below 10, and if in a window of 20 values, more than 3 of them exceed the threshold, then those extremes should be replaced with 'NaN'?
Thank you in advance!

Accepted Answer

Scott MacKenzie
Scott MacKenzie on 5 Mar 2022
There's probably a simpler way to do this, but I believe the code below achieves what you are after:
% test data
x1 = randi([5 12],1,50);
% filtered data in x2
x2 = x1;
% parameters (adjust as necessary)
ws = 6; % window size
t = 10; % threshold
n = 3; % number of points above threshold (must be >n)
for i=ws:length(x1)
idx1 = i-ws+1;
idx2 = i;
idx = (idx1-1) + find(x1(idx1:idx2) > t );
if length(idx)>n
x2(idx) = nan;
end
end
[x1; x2]
ans = 2×50
7 7 7 7 11 9 10 11 10 5 11 10 12 11 9 12 11 11 8 8 5 11 5 10 7 9 5 6 9 6 7 7 7 7 11 9 10 11 10 5 NaN 10 NaN NaN 9 NaN NaN NaN 8 8 5 11 5 10 7 9 5 6 9 6
  2 Comments
Lu Da Silva
Lu Da Silva on 5 Mar 2022
Edited: Lu Da Silva on 5 Mar 2022
The only correction I made is applying the absolute value:
idx = (idx1-1) + find(abs(x1(idx1:idx2)) > t );
Works wonders, thank you!!

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!