any faster way to load part of the mat file?
18 views (last 30 days)
Show older comments
hi: I have a large matfile ROP, with the size of 45*156*300000, but I only need part of the data with given i: ROP(i,:,:).
here I have two ways: 1)directly load the file. 2)use matfile commmand.
below is the code I used, and the tic-toc result:
tic
load ROP
toc
tic
m=matfile('ROP.mat');
test=squeeze(m.ROP(1,:,:));
toc
Elapsed time is 39.358792 seconds.
Elapsed time is 32.416581 seconds.
looks it does not make the loading significantly faster than before. and the only positive effect is that it saves the memory.
when I load a matrix same size with the ROP(1,:,:), it only cost about 3 seconds, which is about 10 times shorter.
so my question is: is there anyway to make the loading faster? while keep the same small memory like matfile command?
thanks!
Li
1 Comment
José-Luis
on 20 Feb 2017
How do you create ROP? The fastest way would probably be to only to save what you need from the start.
Accepted Answer
Guillaume
on 20 Feb 2017
To efficiently use the benefits of matfile it is important to understand how matlab stores matrices in memory (and on file). Matlab stores all the rows of the first column of the first page continuously, then all the rows of the the second column of the first page, etc. until the last column of the first page, then it's the same with the second page. So the storage order is
rop(:, 1, 1), rop(:, 2, 1), ..., rop(:, end, 1), rop(:,1, 2), ..., rop(:, end, end)
With that in mind, using rop(x, :, :) is not efficient at all, since matlab needs to skip size(rop, 1) elements between each element you've requested. However, if you'd requested rop(:, :, x), matlab just needs to seek to the right location and then read the whole lot as one consecutive read. Much faster.
All, this to say, that if you want to speed up matfile read, you ought to change the order of the dimensions when you write that rop variable to file:
rop = permute(rop, [3 2 1]);
save('ROP.mat', 'rop');
m=matfile('ROP.mat');
test = m.ROP(:, :, 1)
2 Comments
Urs Hofmann
on 15 Jul 2019
Edited: Urs Hofmann
on 15 Jul 2019
For me it decreased the time required to read the matrix from 4.9 s to 2.0 s (on top comes 0.5 s for each loading for permuting and 3.5 s while saving).
% hugeMatrix has dimensions [3000, 1001, 1001] of type 'single')
% size: 11.2 Gb
% cropped version: 1.5 Gb
% permuted version
hugeMatrix = permute(hugeMatrix, [2, 3, 1]); % 3.5 sec
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all
m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(:, :, 1:400); % 2.0 sec
hugeMatrix = permute(hugeMatrix, [3, 1, 2]); % 0.5 s
% plain version
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all
m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(1:400, :, :); % 4.9 sec
Meaning: I only save once during a measurement and then load it multiple times, so permuted version is the way to go for me.
More Answers (0)
See Also
Categories
Find more on Workspace Variables and MAT-Files in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!