Why does thingSpeakRead() using numPoints > 100 not always select the latest data?

When I use a value of n greater than 100, my data is read in a fuzzy way, so the most recent data point is not always selcted.

I'm using Apps-React to trigger an Apps MatLab Analysis custom code block.

The architecture is my 'Thing' posts to Channel 1 every 30 seconds. React is triggered by Channel 1 posting. Matlab writes an instruction to 'Control' channel 2. 'Thing' responds to channel 2. Repeat...

I use MATLAB to read Channel 1 and also Channel 2 in order to know its most recent instructions.

The long standing issue I have is with MatLab command thingSpeakRead()

The problem is ctrlTimestamps(last row) is not consistent, in fact the returned dataset depends upon the number of NumPoints.

Here is my test program to prove what happens when I use 101 data points. Yes it returns 101 points BUT the timestamps (posted to status column) are only sometimes in sequence!

% Test to see what value of n causes incorrect read of data
n=101;
%
% Read Input from sensors
%
  readAPIKey = '***'; 
  readChannelID = 1229509; 
% Read raw data channel and map meaningful fields to the raw. 
  [data,timestamps] = thingSpeakRead(readChannelID,'Fields',[1,2,3,4,5,6], ...
                                                'numPoints',(n),'ReadKey',readAPIKey); 
%  display(data,'Sensor data')
%  display(timestamps,'Timestamps')
  rowCountIn = size(data,1);
  display(rowCountIn,'Row count for Input ch ')
%
% Read CONTROL channel for previous settings 
%
  readAPIKey = '***'; 
  readChannelID = 1355058; 
% Read raw data channel and map meaningful fields to the raw. 
  [ctrlData,ctrlTimestamps] = thingSpeakRead(readChannelID,'Fields',[1,2,3,4,5,6,7,8], ...
                                                'NumPoints',(n),'ReadKey',readAPIKey);
%  display(ctrlData,'Control data')
%  display(ctrlTimestamps,'Control timestamps')
%
  rowCount = size(ctrlData,1);
  display(rowCount,'Row count of CONTROL ch ')
  statusUpdate = strcat(' Ctrl time: ',datestr(ctrlTimestamps(rowCount),' HH:MM:SS'));
%
%  Output to CONTROL channel
%
  display (statusUpdate);
  writeAPIKey = '***';
  webwrite('http://api.thingspeak.com/update','api_key',writeAPIKey,'field1',0,'field2',0,'field3',0,'field4',0,'field5',0,'status',statusUpdate);
%  
%
% ----------------END OF PROGRAM ------------
%
Ver -support is: ------ MATLAB Version: 9.13.0.2085925 (R2022b) Update 1 MATLAB License Number: 0 Operating System: Linux 5.4.209-0504209-generic #202208031145 SMP Wed Aug 3 12:39:06 UTC 2022 x86_64 Java Version: Java 1.8.0_292-b10 with AdoptOpenJDK OpenJDK 64-Bit Server VM mixed mode ---------------------------------------------------------------------------------------------------- MATLAB Version 9.13 (R2022b) License 0
----

[Christopher Stapels]

Can you show us the sample data, or some simulated data? I'm not clear exactly what the issue is. It seems that you are saying the problem is that the timestamps returned are out of order?

Do you mean that the most recent datapoint is not always included?

Keep in mind that the ThingSpeak database is eventaully consistant, not exactly real time. Though I would usually expect it to be correct within 30 seconds or better, when running reacts and time controls, there can be scehduling variation involved.

[Austin Jacobs]

Of course, I accidentally forgot to attach output as evidence- duh!

A snippet here with some explanation. I am looking for a Ctrl time that is the previous post i.e. 30 seconds before the created_at timestamp.Where this works I flag it as OK and where it is not, I flag it as outdated. It isn't always alternating as this snippet suggests and it is sometimes longer than a few minutes out of date (see file for more analysed entries):

created_at entry_id status My analysis

2022-11-12T14:47:35+00:00 31561 Ctrl time:14:47:03 OK

2022-11-12T14:48:03+00:00 31562 Ctrl time:14:47:03 Outdated by 30 secs

2022-11-12T14:48:33+00:00 31563 Ctrl time:14:48:03 OK

2022-11-12T14:49:03+00:00 31564 Ctrl time:14:48:03 Outdated by 30 secs

2022-11-12T14:49:34+00:00 31565 Ctrl time:14:49:03 OK

I do get your point that eventual consistency means I have a slightly flawed approach, perhaps. However the key thing is it NEVER goes out of sequence when NumPoints <=100. So this isn't about lag time to write to the underlying database, it is the read from the database when given a larger number of points as a selection criteria that's wonky.

I have started to code a work-round that doesn't use the NumPoints parameter but is time-based using timestamp ('now') as the latest value and I will discover what happens then. Datetime ranges have other nuisance scenarios to cater for, hence I preferred the original method for simplicity.

[Christopher Stapels]

Thanks for the evidence. I can't think of any reason why numbers greater than 100 would change the time recorded by the system when you ingest data. It is possible that the statistical fluctuations in your device posting time get significant when n=100 but I dont think it would be deterministic.

If you want to determine the timestamps exactly, you can use the 'created-at' parameter from your device and write its internal time (assuming the device has a clock).

Consider the following test: you can use ThingSpeakWrite to write 101 points exactly every 30 seconds. Then you would be guaranteed to not have the time lag problem since your points are exactly 30 seconds apart. This seems to prove that is does not have anything to do witht the read command, and it is due to fluctuations in your device writing time. Or I admit I do not understand the issue too well.

There are some pretty weird rules about how reacts are scheduled to be taken care of by the job scheduler - that may also be a part of the issue. I hope you get your process worked out!