Append data to matfile using parallel method
3 views (last 30 days)
Show older comments
Hi:
I have a lots of data that needs to be saved into a test.mat file, below are my test code:
x=rand(10000,1);
save('test.mat','x');
for i=1:1:100
eval(['va_',num2str(i),'=rand(10000,1);'])
eval(['save(','''','test.mat','''',',','''','va_',num2str(i),'''',',','''','-append','''',')'])
end
the problem is that, this is a test code, in my real situation:
1. the number of variables is very large in my situation (up to va_10000).
2. the size of data of each 'va_i' is very large (up to size of 2e6*1).
in this way, although I have upgrade my drive into 960EVO ssd, the saving time is still significantly large.
is there anyway to improve the code into parallel saving? so that I could save the computational cost?
Thanks!
Yu
6 Comments
Walter Roberson
on 13 Sep 2018
Is the size and data type of each variable the same?
Is the data likely to be compressible?
Answers (2)
Steven Lord
on 13 Sep 2018
Consider writing each variable to a different file in such a way that when you want to use them later on you can construct a datastore using that collection of files and make a tall array from the datastore.
Walter Roberson
on 13 Sep 2018
You cannot write to a mat file in parallel. If writing in parallel to a mat file is a requirement then your problem cannot be solved.
If computation of the items is expensive, then do the computation in parallel, writing to different mat files (though potentially one per parallel core rather than one per variable.) Afterwards, merge the files together in a serial loop.
With the data not being compressible, either write in binary or else use the -7.3 option to not compress the output.
2 Comments
Walter Roberson
on 13 Sep 2018
Overall saving time might not increase under the assumption that calculation of the array is expensive. If the average rate of graduation is less than the time required to save one variable then parfor for the calculation and merging afterwards can potentially save time.
Another approach in the case where calculations are expensive is to use a pollable data queue to calculate results in parallel and send them back to the client process to do the saving.
If the average rate of graduation is faster than the time to save one variable then you are probably bandwidth limited in writing to the ssd, and increasing the number of simultaneous writers will not increase the bandwidth.
See Also
Categories
Find more on File Operations in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!