Iteratively save data to mat file

I have to read, process and plot data from 700 text files. I want to save the processed data from one text file to a Mat file before moving on to the next text file. This is a precaution in case of error. I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently.

If I use save('myMatFile.m','x') to save the data from each file, then I will need to create 700 variables in the Mat file; not ideal. If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal. Originally, I use dlmwrite to export to a text file but that is slow.

Also, I'd like to know if my approach is wrong and there is a better way to do this. ANYTHING to speed up this incredibly slow IO process.

Answer 1

[Stephen23]

"I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently."

"If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal."

Use matfile, that is exactly what it was designed for.

[MCM]

When I said, "if I append a single variable in the Mat file then Matlab partially loads it each time; not ideal", I was referring to the matfile function.

When I append using matfile, Matlab gives me a warning that the function will partially read the MAT file in order to do the append

[Walter Roberson]

Consider that if you were changing an existing variable, then MATLAB would have to locate the variable in the file, figure out its size, and make a decision about whether to overwrite in place or to delete the variable and write a new one.

When you append a new variable, MATLAB must first search the .mat file to determine that the variable is not already present, after which it makes whatever changes it needs to the .mat file to write the new variable.

The only way that these processes could be done without partially reading the .mat file would be if every change made by matFile were by simply writing new information on the end without looking inside the file to see what was already there. Not impossible, but it would have the consequence that reading the .mat file would require scanning from the beginning to the end for everything, just in case there was an update later on in the file that overwrote the existing information. It could be done that way, but it would make every read inefficient.

Answer 2

[gonzalo Mier]

First, use the -nocompression option for the save operator if you have a lot of info.

Maybe you could improve your performance writing the info in 700 different mat files as save("mat_"+num2str(i)+".m") so you don't have to read the info each time.

[Walter Roberson]

I recently posted a timing test using matFile with and without compression. For the rand() that the user was working with, turning off compression made a big difference. It is plausible that in some other cases like a mostly 0 matrix that compression might be more efficient due to reduced disk i/o.

Be sure to save with -v7.3 to use matFile efficiently