Filtering Sensor Data to Remove Temperature Ramp Effects

Hi,

I need to apply some filtering to my data in order to remove samples collected while the temperature is ramping up, and to use only the samples where the temperature is relatively stable.

There will still be some small fluctuations in temperature during these stable periods, but they should be minimal. As shown in the attached plots, the regions highlighted with red circles in the top plot correspond to the temperature ramp and settling areas that I would like to filter out. The bottom plot shows the sensor signal, where these temperature changes clearly introduce ripple and transient effects.

Is there a recommended function or method to reliably detect and filter out these ramping regions, leaving only the stable temperature data for further analysis?

[dpb]

Attach a dataset for folks to play with...

[Dharmesh]

@dpb Please find it attached

[dpb]

Have you tried findchangepts to see if it can detect the slope changes for you? It may not be sufficiently sensitive, but worth a shot.

The next obvious step is to look at first/second derivatives...

[Dharmesh]

No, i will give it a try

Answer 1

[dpb]

load SensorLog_ALLS2
%head(Tall)
Tall.Properties.VariableNames=strrep(Tall.Properties.VariableNames,' ','');  % remove blanks
Tall.Properties.VariableNames(2:3)={'T','RH'};                               % more convenient
head(Tall)

         Timestamp          T       RH      NOWE     NOAUX    O3WE     O3AUX    NO2WE    NO2AUX                    SourceFile                
    ___________________    ____    _____    _____    _____    _____    _____    _____    ______    __________________________________________

    2025-10-30 11:04:54    8.42    57.01    0.135    0.172    0.232    0.235    0.255    0.257     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:04:59    8.37    56.84    0.135    0.173    0.232    0.235    0.256    0.256     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:05:04    8.33    56.67    0.136    0.174    0.233    0.235    0.256    0.258     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:05:09    8.29    56.51    0.137    0.174    0.233    0.236    0.257    0.256     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:05:14    8.25    56.34    0.139    0.174    0.234    0.236    0.257    0.256     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:05:19    8.21    56.18    0.139    0.176    0.234    0.237    0.257    0.255     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:05:24    8.17    56.02     0.14    0.179    0.234    0.237    0.258    0.254     "SensorLog_Sensor-002_20251030_110449.csv"
    2025-10-30 11:05:30    8.13    55.86    0.141     0.18    0.235    0.238    0.258    0.255     "SensorLog_Sensor-002_20251030_110449.csv"

findchangepts(Tall.T,'Statistic','linear')      % look at overall

●

Pstart=500; Pend=2500;                                      % pick start, end locations
findchangepts(Tall.T(Pstart:Pend),'Statistic','linear')     % look at inner section only

N=18;
findchangepts(Tall.T(Pstart:Pend),'Statistic','linear', ...
                                   'MaxNumChanges',N)      % and make sensitive

That doesn't look too bad for starters. Will need to add some heuristics to guess where to start/stop and how many steps there are.

I wish there were some other way to influence its sensitivity besides just the number of points so could make it based on the actual change in computed values.

You might find fitting the linear expression to the total and looking at residuals a helpful approach-- that's basically what findchangepts is doing.

Note I cheated by counting the number of intervals and then set N to be 2X that number in order to delineate the start/stop sections. An earlier post with N=10 wasn't enough so several were midpoints and only one break between sections instead of two.

Using the two-output form and using the returned residual in a loop as N is increased in steps of 2 might work pretty well.

[Dharmesh]

Thanks I will review this process and pass on any comments.

The aim is eventually prepare a Look up table for each temperture bucket 10 degree steps, so i need to clean up the data and make it more stable

[dpb]

If the temperatures are controlled setpoints so you know where the plateaus are, that should make setting the number pretty easy.

Another ploy would be to find those values within some error band about those setpoint values.

[Dharmesh]

it seems that it needs signal processing toolbox?

[dpb]

which -all findchangepts

/MATLAB/toolbox/signal/signal/findchangepts.m

is in Sitnal Processing TB, yes....without, you would need to "roll your own" similar function; the starting point would be still picking the two ends of the transient you want to cover and fitting a straight line, then looking at the residuals to find where they have larger excursions to locate smaller subsections. "Wash, rinse, repeat..."

As above, if you do know the setpoint levels, then locating regions within an ever tightening tolerance of those and detrending those sections should also help or might be even more efficient.

[Dharmesh]

Yes, the ideal set points can be 0, 10, 20, 30, 40, and 50 °C (possibly 48 °C if we are not able to reach 50 °C reliably).

I have been visually reviewing the data, and if we only use data samples for each sensor at these set points, we could define a window of ±1 °C and calculate the mean of all samples within that temperature range.

There is a large amount of data, much of which can be filtered out using this approach.