What is the fastest and smartest way to import and manage/plot many text files in matlab?
39 views (last 30 days)
Show older comments
Giuseppe
on 8 Feb 2022
Commented: Walter Roberson
on 17 Feb 2022
Hi guys! I've processed my data in Fortran and I produced 55 .txt files (as shown in the following image). These files contains asteroids position stored in three columns (x,y,z coordinates); see an example file atteched below.
I'm looking for a smart and quick procedure to import them in matlab by using few code lines. Then I want to plot my data and maybe perform further computations. My idea is to store the files in a data structure subdivided by the number of asteroids (55 in my case), then create a loop that runs through all the files in my folder and maybe be able to stop at the last file by automatically determining the index value of said loop. In a few words I would like to create an automated procedure to import many files in a single variable that I can then manage for further calculations or plots in matlab. I found a code that seems to fit my request:
filenames = dir('D:\OneDrive\MSc_Thesis\Projects\NEOs_orbits\OutputFiles\Orbit_asteroid_*.txt');
number_of_files = numel(filenames); col_values = [];
for ii = 1:number_of_files
all_values = load(filenames(ii).name); %Error using load Unable to read file 'Orbit_asteroid_01.txt'. No such file or directory.
col_values = [col_values; all_values(:,1)];
end
But I get the error printed as comment. Can you help me, please?
0 Comments
Accepted Answer
Walter Roberson
on 9 Feb 2022
Edited: Walter Roberson
on 9 Feb 2022
filename = 'https://www.mathworks.com/matlabcentral/answers/uploaded_files/888225/Orbit_asteroid_02.txt';
%"warm up" -- this will likely read the file into memory so we can remove
%the disk speed component
for K = 1 : 3; urlread(filename); end
%done warm-up -- mostly ignore the above, it is disk dominated
tic;
allvalues = readmatrix(filename);
toc
tic;
allvalues = table2array(readtable(filename));
toc
tic;
S = urlread(filename);
allvalues = cell2mat(textscan(S, '%f %f %f', 'HeaderLines', 3));
toc
tic;
L = regexp(urlread(filename), '\r?\n', 'split');
L(1:3) = [];
allvalues = cell2mat(cellfun(@(S) sscanf(S, '%f %f %f', [1 inf]), L, 'uniform', 0));
toc
tic;
L = regexp(urlread(filename), '\r?\n', 'split');
L(1:3) = [];
S = strjoin(L, '\n');
allvalues = sscanf(S, '%f %f %f', [3 inf]).';
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
allvalues = sscanf(L, '%f %f %f', [3 inf]).';
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
allvalues = cell2mat(textscan(L, '%f %f %f'));
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
allvalues = str2num(L);
toc
tic;
L = regexprep(urlread(filename), '^.*\n.*\n.*\n', '', 'once', 'dotexceptnewline');
T = tempname();
fid = fopen(T, 'w');
fwrite(fid, L);
fclose(fid);
allvalues = load(T);
toc
I had a bug in the earlier version of some of the conversions, so these conclusions are revised:
Fastest: read the file as text, use text processing to remove the header, textscan() the result. Oddly this was notably faster than textscan() of the full data with 'headerliens'
Second fastest: various text processing methods are close enough to each other over multiple runs that I cannot judge between them. That is, using this test facility, the times varied enough that the answer is not clear. Running on your own system would likely produce a different result.
Of particular note: read the file as a string, removing the header, writing the file out, and using load(), is still about 5 times faster than using readtable()
In your real system, it might turn out that using fopen() to open the file, and then use textscan(), might work out as the fastest. It is a bit difficult to benchmark for the case where the file is not already in filesystem cache, without risking the possibility that as you run the several tests, that the second and following tests might be taking advantage of the file system already having cached the file.
3 Comments
Walter Roberson
on 9 Feb 2022
struct is certainly do-able.
projectdir = 'D:\OneDrive\MSc_Thesis\Projects\NEOs_orbits\OutputFiles';
dinfo = dir( fullfile(projectdir, 'Orbit_asteroid_*.txt') );
filenames = fullfile( {dinfo.folder}, {dinfo.name});
number_of_files = numel(filenames);
Astroid(number_of_files) = struct('x', [], 'y', [], 'z'); %pre-allocates
for ii = 1:number_of_files
thisfilename = filenames{ii};
all_values = CodeToLoadTheFile(thisfilename);
Astroid(ii).x = all_values(:,1);
Astroid(ii).y = all_values(:,2);
Astroid(ii).z = all_values(:,3);
end
but cell array might have higher performance, perhaps
projectdir = 'D:\OneDrive\MSc_Thesis\Projects\NEOs_orbits\OutputFiles';
dinfo = dir( fullfile(projectdir, 'Orbit_asteroid_*.txt') );
filenames = fullfile( {dinfo.folder}, {dinfo.name});
number_of_files = numel(filenames);
Astroid = cell(number_of_files, 1);
for ii = 1:number_of_files
thisfilename = filenames{ii};
all_values = CodeToLoadTheFile(thisfilename);
Astroid{ii} = all_values;
end
which would
plot3(Asteroid{ii}(:,1), Asteroid{ii}(:,2), Asteroid({ii}(:,3))
More Answers (2)
David Hill
on 8 Feb 2022
As long as you are in the folder you want, you just need the file name.
for k=1:55
f=num2str(k);
if length(f)==1
f=strcat('0',f);
end
r(:,(k-1)*3+1:3*k)=readmatrix(strcat('Orbit_asteroid_',f));
end
%then plot what you want
plot3(r(:,4),r(:,5),r(:,6));%2nd asteroid
0 Comments
Highphi
on 8 Feb 2022
you can't use 'load' here, you can only load MAT-files or certain ASCII files
best way to import these files is by using:
all_values = readtable(filenames(ii).name);
all_values = table2array(all_values);
Good luck
1 Comment
Walter Roberson
on 17 Feb 2022
Yes and no.
You are correct that the original format of the file is something that load() would not process. And in cases where clarity and convenience is important (which is a lot of the time, really), using one of the other possibilities instead of load() would be a better choice.
But as I showed near the end of https://www.mathworks.com/matlabcentral/answers/1645930-what-is-the-fastest-and-smartest-way-to-import-and-manage-plot-many-text-files-in-matlab#answer_891795 it is possible to read the file, strip off the header, write the the revised content, and load() the revised content, and still end up with a processing time that is competitive with nearly all the more convenient approaches. At least until the file starts taking a major fraction of your memory, or until the hard-drive speed starts to dominate.
See Also
Categories
Find more on Database Toolbox in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!