Mutiple Time series synchronization

Hi

I want to synchronize multiple time series

Can any suggest how to do it.

Thanks

[Jogger]

I want new time series close to 6 min duration

>> firmware_FAFTSetup_DEBUG.Content.Date(end)-firmware_FAFTSetup_DEBUG.Content.Date(1)

ans = 

  duration

   00:06:09

>> servo_v4_uart.Content.Date(end)- servo_v4_uart.Content.Date(1)

ans = 

  duration

   00:06:10

>> ec_uart.Content.Date(end)- ec_uart.Content.Date(1)

ans = 

  duration

   00:06:03

>> cr50_uart.Content.Date(end)- cr50_uart.Content.Date(1)

ans = 

  duration

   00:05:25

>> firmware_FAFTSetup_INFO.Content.Date(end)- firmware_FAFTSetup_INFO.Content.Date(1)

ans = 

  duration

   00:06:09

>> 

[Guillaume]

The synchrosize function (for timeseries) has several synchronisation methods, which one do you want to use?

Note that synchronisation does not affect the duration of a series, it only changes at which period that series is sampled.

If you want to make a time series longer (in period, not number of samples) then you will have to extrapolate. If you want to make it shorter, you'll simply have to clip it.

[Jogger]

Thanks .

I prefer to do extrapolate . Can you please elaborate more with an example from attached data ?

I tried following but I am getting error

>> ts1 = firmware_FAFTSetup_DEBUG.Content.Date;
>> ts2 = firmware_FAFTSetup_INFO.Content.Date;
>> synchronize(ts1,ts2,'Uniform','Interval',.1)
Undefined function 'synchronize' for input arguments of type 'datetime'.

[Guillaume]

Right, so you don't have timeseries. Perhaps, firmware_FAFTSetup_DEBUG is a timetable? Or a table? Or maybe a plain structure? In any case, it's important to use the correct term.

In order to extrapolate, you need a fit for your data. What sort of fitting function are you planning to use?

[Jogger]

I prefer Uniform for data fitting.

[Guillaume]

The synchronisation method applies to the timing of your data, not to the data itself. You're not going to extrapolate a sinusoid the same way you extrapolate a line.

[Jogger]

Yes I want to synchronize the data timing. but i am lost on how to proceed

I meant synchronization would result a uniform interval of New timeseries from the my attached input data.

[Jogger]

My goal is to get a common time series with text data. If for a time instance data is present then choose else skip particular data.look for other data if present on time then put that data.Like this I want to have a common time series for from different readtable.

[Guillaume]

My goal is to get a common time series with text data

If it's textual data, I have no idea why you were asking about extrapolation. How do you extrapolate text?

Anyway, to synchronize several timetables you'd simply do it in a loop. I can show you how it's done once I understand what the inputs are. So far, you seem to have shown date variables, a timeseries or timetable needs date and data to be useful.

I'm not even sure what kind of data you have, so please state whether it's timeseries, timetable, table or something else.

Answer 1

[Guillaume]

I think this does what you want:

contentfields = fieldnames(Content);
for fieldidx = 1:numel(contentfields)    %iterate over each field of Content
    structtable = struct2cell(Content.(contentfields{fieldidx}));  %extract the structure within the field by converting it to cell array (avoids having to work out what its name is)
    structtimetable = table2timetable(structtable{1}(:, {'Date', 'Message'}));  %convert to timetable, only keeping Date and Message
    structtimetable.Properties.VariableNames{1} = sprintf('Message_%s', contentfields{fieldidx});  %rename Message variable so we know where it came from
    structtimetable = rmmissing(structtimetable);  %remove invalid rows to avoid problems with synchronize
    if fieldidx == 1
        joinedtimetable = structtimetable;  %1st time, create output
    else  
        joinedtimetable = synchronize(joinedtimetable, structtimetable, 'union'); %subsequent times, synchronize. Choose whichever method is prefered
    end
end

The main issue is that some (all?) of your tables contain rows with NaT which cause problem with synchronisation. You may want to investigate that. I've removed these rows here.

[Jogger]

Super ! Yes,NaT was a big problem, It destroyed complete sync process. Many thanks for your kind help.

[Jogger]

Hi Guillaume

I had a relook into the date timeseries.

If you look time content and message content they are NOT aligned properly.

I mean if col1 is time and col2, col3 etc are message then we have NOT sorted the message according to time.

This is wrong

  '06/19 18:39:25.000'  ''  ''  '>'  '> chan save'  ''  '' 
   & so on.     
  '08/19 10:06:37.000'  ''  ''  ''  ''  'Called: system.is_available()'  '' 
  '08/19 10:06:38.000'  ''  ''  ''  ''  'Called: system.dump_log(True,)'  '' 
  I see time dip
  '06/24 18:39:27.300'  'Setting usb_mux_oe1 to ''on'''  'Setting usb_mux_oe1 to ''on'''  ''  ''  ''  '' 

What I expect is Timestamp should increase ( ascending order ) & message to synchronize accordingly.

Example in right order i.e. each colomn should follow timeseries & puts message in time sequential order

  '06/19 18:39:25.000'  ''  ''  '>'  '> chan save'  ''  '' 
  '06/24 18:39:27.300'  'Setting usb_mux_oe1 to ''on'''  'Setting usb_mux_oe1 to ''on'''  ''  ''  ''  '' 
  '08/19 10:06:37.000'  ''  ''  ''  ''  'Called: system.is_available()'  '' 
  '08/19 10:06:38.000'  ''  ''  ''  ''  'Called: system.dump_log(True,)'  '' 

I want to align the timeseries according to duration. No need of MM/dd .Can you please help

[Jogger]

Please refer spread sheet how the data should look like

[Rik]

Comment posted as flag by sriram shastry:

Synchronization is not working as expected.Please help me.

[Jogger]

This will work if we remove the day,month part and keep Hour,minute ,seconds and milliseconds section

ddMM - remove

HH:mm:ss.SSS keep.

This is sufficient for me

[Guillaume]

I'm sure we discussed this before. You have a strange concept of time. A date is a point in time, a duration is the distance between two dates and to me it doesn't make sense to take one part of a date and say it's a duration. I could understand subtracting the first date of a table from all the other dates in the same table to get a duration (time ellapsed since the start of the log) but just taking the hour/minute/second part of a date and calling that a duration makes no sense to me.

Anyway, if that's what you want, it's trivial to do:

contentfields = fieldnames(Content);
for fieldidx = 1:numel(contentfields)    %iterate over each field of Content
    structtable = struct2cell(Content.(contentfields{fieldidx}));  %extract the structure within the field by converting it to cell array (avoids having to work out what its name is)
    structtable = structtable{1}; %get the table out of the cell
    structtable.Date = duration(structtable.Date.Hour, structtable.Date.Minute, structtable.Date.Second);  %extract h/m/s and make that a duration. This makes no sense to me!
    structtable.Properties.VariableNames{1} = 'WeirdDuration';  %rename the time column
    structtimetable = table2timetable(structtable(:, {'WeirdDuration', 'Message'}));  %convert to timetable, only keeping Date and Message
    structtimetable.Properties.VariableNames{1} = sprintf('Message_%s', contentfields{fieldidx});  %rename Message variable so we know where it came from
    structtimetable = rmmissing(structtimetable);  %remove invalid rows to avoid problems with synchronize
    if fieldidx == 1
        joinedtimetable = structtimetable;  %1st time, create output
    else  
        joinedtimetable = synchronize(joinedtimetable, structtimetable, 'union'); %subsequent times, synchronize. Choose whichever method is prefered
    end
end

[Jogger]

I could understand subtracting the first date of a table from all the other dates in the same table to get a duration

If you can give the implementation along with milliseconds information it would be very helpful

Perfect! Thanks a lot ! I get the data as expected.

[Guillaume]

Note that subtracting the first date of a table from all the other dates in the same table to get a duration is not what I have done above. I've just extract the hour/minute/seconds and made that a duration, as you asked (even though it makes no sense to me).

If you do want to subtract the first date to convert to duration then change the line:

structtable.Date = duration(structtable.Date.Hour, structtable.Date.Minute, structtable.Date.Second);  %extract h/m/s and make that a duration. This makes no sense to me!

to

structtable.Date = structtable.Date - structtable.Date(1);  %elapsed time since start of log.

[Jogger]

Thanks ! but I was wondering where milliseconds count is gone ?

[Guillaume]

Nowhere probably. If it's not visible, it's most likely because the default duration format doesn't display it. You can easily change the Format of the variable to whatever you want:

joinedtimetable.RowTimes.Format = 'hh:mm:ss.SSS'; 

[Jogger]

Sure! Thanks