Clear Filters
Clear Filters

Data Source for downloaded data from an HTTP API advice

1 view (last 30 days)
Hi
I am downloading data from any HTTP API. As shown in code below:
for i = 1:datelistsize(2)
disp([datelist.start(i),datelist.stop(i)]);
live_data_downloaded_data_temp =device_data(deviceid,datelist.start(i),''+datelist.stop(i));
live_data_downloaded_data =[live_data_downloaded_data live_data_downloaded_data_temp'];
end
save("deviceid_"+deviceid,'live_data_downloaded_data','ref_data_timestamp','ref_data_sensordata')
As time progresses i do not want to be re downloading data that i alread have.
If i am correct i could use datasource.
I would need to store the last times stamp of my data, then what i aim to do is on every instance i call my code, i would save "live_data_downloaded_data" into a new .mat file. Then when i need to read the data, i would have multiple files.
Does anyone have example how i would need to modify my code, so a data source can acklowedge it?

Accepted Answer

Adithya
Adithya on 22 May 2023
To implement a data source that acknowledges previously downloaded data and avoids re-downloading it, you can use the concept of "checkpoints". It allows you to save the state of your data retrieval process, so you can resume from where you left off instead of starting from the beginning. Here's an example of how you can modify your code to incorporate checkpoints:
deviceid = 'your_device_id';
checkpointFile = 'checkpoint.mat'; % File to store the checkpoint data
% Load previous checkpoint if it exists
if exist(checkpointFile, 'file')
load(checkpointFile);
else
checkpoint = 1; % Starting checkpoint (first data point)
end
for i = checkpoint:datelistsize(2)
disp([datelist.start(i), datelist.stop(i)]);
% Download data only if it hasn't been previously downloaded
if i >= checkpoint
live_data_downloaded_data_temp = device_data(deviceid, datelist.start(i), ''+datelist.stop(i));
live_data_downloaded_data = [live_data_downloaded_data live_data_downloaded_data_temp'];
end
checkpoint = i + 1; % Update the checkpoint to the next iteration
% Save the checkpoint data
save(checkpointFile, 'checkpoint');
end
% Save the final downloaded data
save("deviceid_" + deviceid, 'live_data_downloaded_data', 'ref_data_timestamp', 'ref_data_sensordata');
In this modified code, a "checkpoint" is introduced to keep track of the last successfully downloaded data point. The checkpoint value is stored in a "checkpoint" file (checkpoint.mat) to persist the state between different instances of running the code.
When you run the code, it checks if a "checkpoint" file exists. If it does, it loads the checkpoint value, indicating the last successful iteration. The code then continues from that checkpoint, downloading the remaining data points.
The downloaded data is only fetched if the current iteration is at or beyond the checkpoint value. This ensures that previously downloaded data is not re-downloaded.
After each iteration, the "checkpoint" value is updated to the next iteration, and the checkpoint file is saved to store the progress.
Finally, the final downloaded data is saved along with any other required data.
By using this approach, you can resume data retrieval from the last successful iteration and avoid redundant downloads in subsequent runs of the code.

More Answers (0)

Categories

Find more on Downloads in Help Center and File Exchange

Products


Release

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!