Clear Filters
Clear Filters

MATLAB Tools to Evaluate and Mitigate Sensor Noise

12 views (last 30 days)
Hi,
I possess a dataset representing the output of an electronic NO2 chemical sensor, which I've compared against reference data. I've been utilizing the Regression Learning App in MATLAB to devise an appropriate model for predictions, and it has proven effective. However, like many electronic sensors, there's noticeable noise, particularly evident when the gas concentration is minimal, e.g., 10-15 ppb.
I'm eager to further enhance my overall process beyond just using the model from the Regression Learning App. Are there any additional tools or features in MATLAB that could help me evaluate and possibly mitigate this noise when concentration levels are low? I'm open to specific tools or any machine learning tools that might discern patterns in the data, subsequently refining my results.
  5 Comments
Star Strider
Star Strider on 18 Oct 2023
If resampling the data removes some of the necessary information, then it is likely best not to resamjple it before processing it.
One effective way of dealing with noise while retaining the underlying signal is to use the sgolayfilt function. There is not any one good way of dealing with noise, however that would be my first choice, especially since I do not know the details.
Another way of reducing the noise while keeping the details is to create specific segments of the data called ensembles, ideally centred on the peaks, and then processing them individually or by averaging them first and then processing them.
Since I am not certain what you want from the data, that is all I can suggest at present.
Dharmesh Joshi
Dharmesh Joshi on 19 Oct 2023
Thanks
I will give that a go. For your reference i have attached the data.
There is the raw data, hour average, and ref data.
I need to improve the hour average data to include the peaks. Using the ref data, you can get an idea where the peaks would be in the raw data.

Sign in to comment.

Accepted Answer

Star Strider
Star Strider on 19 Oct 2023
I am not certain that I understand the sort of processing you are doing. Some of the signal noise is broadband, however much of it can be eliminiated either using a Savitzky-Golay filter or a simple lowpass filter.
I did both here —
LD = load('dataFilter.mat')
LD = struct with fields:
hour_average: [5554×1 timetable] raw_min_data: [333239×1 timetable] refdata: [5555×1 timetable]
rmd = LD.raw_min_data
rmd = 333239×1 timetable
Time Var1 ____________________ _______ 16-Jan-2023 13:02:00 2.9424 16-Jan-2023 13:03:00 2.1115 16-Jan-2023 13:04:00 1.7951 16-Jan-2023 13:05:00 2.072 16-Jan-2023 13:06:00 2.9067 16-Jan-2023 13:07:00 1.5364 16-Jan-2023 13:08:00 2.0293 16-Jan-2023 13:09:00 1.6482 16-Jan-2023 13:10:00 1.6591 16-Jan-2023 13:11:00 5.8567 16-Jan-2023 13:12:00 0.32883 16-Jan-2023 13:13:00 0.62192 16-Jan-2023 13:14:00 0.915 16-Jan-2023 13:15:00 3.7207 16-Jan-2023 13:16:00 0.5806 16-Jan-2023 13:17:00 0.2533
Ts = mean(diff(rmd.Time))
Ts = duration
00:01:00
[h,m,s] = hms(Ts);
Fs = 1/60; % Hz
Fn = Fs/2;
CheckSampling = nnz(diff(rmd.Time) > minutes(1))
CheckSampling = 0
sgf = sgolayfilt(rmd.Var1, 3, 501);
figure
plot(rmd.Time, rmd.Var1, 'DisplayName','Original Signal')
hold on
plot(rmd.Time, sgf, '-r', 'DisplayName','Filtered Signal (‘sgolayfilt’)')
hold off
grid
xlim([rmd.Time(1) rmd.Time(1)+caldays(7)])
xlabel('Time (minutes)')
ylabel('Value')
legend('Location','best')
L = size(rmd,1);
NFFT = 2^nextpow2(L);
FTs = fft((rmd.Var1-mean(rmd.Var1)).*hann(L), NFFT)/L;
Fv = linspace(0, 1, NFFT/2+1).'*Fn;
Iv = 1:numel(Fv);
[pks,locs] = findpeaks(abs(FTs(Iv))*2, 'MinPeakProminence', 0.2);
Major_Peaks = table(Fv(locs), pks, 1./(Fv(locs)*3600*24), 'VariableNames',{'Frequency (Hz)','Peak Amplitudes','Period (Days/Cycle)'})
Major_Peaks = 3×3 table
Frequency (Hz) Peak Amplitudes Period (Days/Cycle) ______________ _______________ ___________________ 6.3578e-08 0.73208 182.04 1.1603e-05 0.37121 0.9975 2.3142e-05 0.30513 0.50012
figure
plot(Fv, abs(FTs(Iv))*2)
hold on
plot(Fv(locs), abs(FTs(locs))*2, '^r')
hold off
grid
xlim([0 0.00005])
xlabel('Frequency (Hz)')
ylabel('Magnitude)')
xline(3E-5, '--k', 'Lowpass Filter Cutoff Frequency')
Var1_filt = lowpass(rmd.Var1, 3E-5, Fs, 'ImpulseResponse','iir');
figure
plot(rmd.Time, rmd.Var1, 'DisplayName','Original Signal')
hold on
plot(rmd.Time, Var1_filt, '-r', 'DisplayName','Filtered Signal (‘lowpass’)')
hold off
grid
xlim([rmd.Time(1) rmd.Time(1)+caldays(7)])
xlabel('Time (minutes)')
ylabel('Value')
legend('Location','best')
Ths sort of processing you do depends on the details you want to model. I hope my analysis provides some insight.
.
  22 Comments
Star Strider
Star Strider on 30 Oct 2023
If it is the same issue, it would likely be best to continue it there. I will watch for it.
Please explain where we are in this effort now, since I have lost track.

Sign in to comment.

More Answers (0)

Products


Release

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!