MATLAB Answers

how to append data from mat file.?

5 views (last 30 days)
pruth
pruth on 15 Feb 2016
Answered: dpb on 16 Feb 2016
hello. guys... my data is so heavy(4-5 gb data)...it taking so much time for loading ... i have some mat files...they are actuality cell array...files are like brodata.mat <5136 *2>, brodata1.mat <4328*2>,brodata2.mat <4367*2>,brodata3.mat <5328*2>..... i am trying to make only one file (cell array) ..which contain all the variables from these files i.e <19159 *2 >.
  2 Comments
pruth
pruth on 16 Feb 2016
in each cell array there are two column , like bellow
<15094x3 double> 735965.041666667
<15072x3 double> 735965.083333333
<15096x3 double> 735965.166666667
<15078x3 double> 735965.250000000
<15101x3 double> 735965.291666667
<15070x3 double> 735965.375000000
<15097x3 double> 735965.458333333
.
.
.
<15106x3 double> 736145.625000000
<15137x3 double> 736145.708333333
<15111x3 double> 736145.791666667
<15133x3 double> 736145.833333333
<15132x3 double> 736145.916666667
i want one file which contain all the data from all cell array. i want next cell arrays two column should append here below these two columns. hope u understand. i m little slow in English.

Sign in to comment.

Answers (1)

dpb
dpb on 16 Feb 2016
I expect you'll just make your loading time worse but I'd use the matfile object to try this and see how it goes...
Start by making a copy of the first file in a new file so have a way to recover if having a huge file isn't the answer to your problems...
copyfile('brodata.mat','newbro.mat') % make the initial new file
newObj=matfile('newbro.mat','Writeable','true'); % create matfile object as writeable
d=dir('brodata*.mat'); % get the list of files
for 2=1:length(d) % process; skip the base (dir returns ordered list)
data=load d(i).name; % get the data from the file
newObj.data=[newObj.data;data];
clear data % get rid of extra copy
end
In all likelihood the above will run into memory issues as it does create a copy in memory
"data" above is a placeholder for your cell variable name in your files; use whatever it is.
You can -append to a .mat file, but only additional variables with save; if you use the same variable name it replaces that name, it does not add onto that array internally -- illustration:
>> whos -file new.mat
Name Size Bytes Class Attributes
c 4x2 6912 cell
>> c
c =
[100x2 double] [7.3638e+05]
[100x2 double] [7.3638e+05]
>> save new.mat c -append
>> whos -file new.mat
Name Size Bytes Class Attributes
c 2x2 3456 cell
>>
Net result is the 2x2 from memory replace the already existing 4x2 on file.
Only way I can think of to create larger arrays on file w/o holding in memory would be to create stream files (which don't support cell arrays well). Then you could do simple file system copy but you lose the existing data structure.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!