Filling gaps in time series with Nan

28 views (last 30 days)
Hi there!
I^m currently working with several series of data from different sensor.Each sensor is measuring different quantities, and, theoretically, should provide a value each 30s, so that in 1 day I should have 2880 values for each sensor. I have a matrix for each sensor with 2 columns: timestamp and the corresponding measured data. The problem is that sometime some sensors just misses the measure (very randomly) and they do not record gaps as anything, they simply skip to the next measure, so the length of the matrixes is never 2880, but always shorter and different between the sensors. What i would like to do is to detect these gaps in the timestamp column and to insert Nan values in the corresponding data column. If anyone could offer any suggestions it would be very much appreciated!
PS: the timestamp column is already in Matlab time format: (2013-03-01=735294) (30s=-3.4722e-004)

Accepted Answer

José-Luis on 17 May 2013
Edited: José-Luis on 17 May 2013
Sounds like a job for intersect():
%Generating random dataset:
numVals = 2000;
all_seconds = 0:30/86400:1;
all_seconds(end) = [];
all_seconds = all_seconds' + floor(now);
your_seconds = all_seconds(sort(randperm(numel(all_seconds),2000)));
your_data = [your_seconds rand(numVals,1)];
%Actually doing what you asked:
filled_data = [all_seconds NaN(numel(all_seconds),1)];
[bla ia ib] = intersect(filled_data(:,1),your_data(:,1));
filled_data(ia,2) = your_data(ib,2);
Please accept an answer if it helps you.

More Answers (1)

Matt Kindig
Matt Kindig on 17 May 2013
Edited: Matt Kindig on 17 May 2013
One way that considers floating point errors:
startValue = datenum(2013,5,17,0,0,0); %your starting observation
endValue = datenum(2013,5,17,23,59,59); %your ending observation
increment = datenum(2013,1,1,0,0,30)-datenum(2013,1,1,0,0,0); %30s increments
timePoints = startValue:increment:endValue; %desired time points
% timestamps is your existing time points
% values is your existing sensor measurements
% to illustrate, we will just remove some random time points
ix = sort(ceil(rand(1,10)*length(timePoints)));
timestamps = timePoints;
timestamps(ix) = [];
values = rand(size(timestamps)); %random values
%now get good points
n= histc(timestamps, [timePoints-(increment/3), timePoints(end)+(increment/3)]);
% n will be 1 if sample time is present, zero otherwise.
allValues = NaN(size(timePoints)); %initially set to NaN
allValues(n==1) = values; %fill in with "real" data

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!