MATLAB Answers

Import files based on file name

4 views (last 30 days)
I have a folder containing around 300,000 files. I don't need to import all the files.
Problem: How can I import the files based on specific file name?
Problem example:
In the picture below I have a section of the files, which are all in the same folder. I only want to import the .data files. But I don't need all the .data files to be import only the last one of every serie.
Up until now I have the following code:
files = dir(fullimpfile(pwd,'*.data*'));
expData = cell(length(files),1);
for i = 1:length(files)
fid = fopen(fullfile(files(i).folder,files(i).name),'r');
%% Reading the data
% Read all the data from the file
dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
end
This code imports all of the data files. How can I use I think a for loop to only import only the last .data files of every serie?

  0 Comments

Sign in to comment.

Accepted Answer

Stephen Cobeldick
Stephen Cobeldick on 2 Sep 2020
fnm = {...
'LedgeTest_muSP_0.10_muRP_0.10.1.data.0',...
'LedgeTest_muSP_0.10_muRP_0.10.1.data.1',...
'LedgeTest_muSP_0.10_muRP_0.10.1.data.2',...
'LedgeTest_muSP_0.10_muRP_0.10.1.data.21',...
'LedgeTest_muSP_0.10_muRP_0.20.1.data.0',...
'LedgeTest_muSP_0.10_muRP_0.20.1.data.1',...
'LedgeTest_muSP_0.10_muRP_0.20.1.data.2',...
'LedgeTest_muSP_0.10_muRP_0.20.1.data.11'}
spl = regexp(fnm,'\.data\.','split','once');
spl = vertcat(spl{:});
vec = str2double(spl(:,2));
[~,idx] = sort(vec);
[~,idy,idz] = unique(spl(idx,1),'last');
out = fnm(idx(idy))
Giving:
out =
'LedgeTest_muSP_0.10_muRP_0.10.1.data.21'
'LedgeTest_muSP_0.10_muRP_0.20.1.data.11'
Use it like this:
D = pwd;
tmp = dir(fullfile(D,'*.data.*'));
fnm = {tmp.name};
...
for k = 1:numel(out)
fid = fopen(fullfile(D,out{k}),'r');
...
end

  9 Comments

Show 6 older comments
Tessa Kol
Tessa Kol on 25 Sep 2020
Thank you for your explenation!
Is it also possible to do the same approach, but then the files are in different folders. I tried that with the followin piece of code:
%% Loading the data
rhoPart = 2540;
% Select the main folder
Folder = uigetdir;
% Find all .data files in the sub folders
files = dir(fullfile(Folder,'\**\*.data*'));
% Select only .data file from the last time step of each simulation run
spl = regexp({files.name},'\.data\.','split','once');
spl = vertcat(spl{:});
vec = str2double(spl(:,2));
[~,idx] = sort(vec);
[~,idy,idz] = unique(spl(idx,1),'last');
out = {files(idx(idy)).name};
k = 1;
for i = 1:length(out)
fid = fopen(fullfile(files(i).folder,out{i}),'r');
%% Reading the data
% Read all the data from the file
dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
% frewind(fid);
% Write headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
% runData{k} = strsplit(fgetl(fid), ' ');
% Write only the x, y, and z components of the particles, particle radius,
% z component+ particle radius and volume of the particle
expData{k} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
% Write only the vx,vy,vz of the particles and magnitude
velData{k} = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
fclose(fid);
k = k + 1;
end
But this obviously doesn't work the files(i).folder doesn't match the out{i}.
The main folder where all the .data files are stored is called 2Dtest4.
Then, I have 4 subfolders called:
2Dtest_all-0.45
2Dtest_all-0.67
2Dtest_all-0.89
2Dtest_all-0.123
I those subfolders are the .data files stored.
Tessa Kol
Tessa Kol on 25 Sep 2020
files(idx(idy)).folder
Did not work either. I got an error saying:
Error using textscan
Invalid file identifier. Use fopen to generate a valid file identifier.
Error in LedgeTest2D_results (line 33)
dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
Stephen Cobeldick
Stephen Cobeldick on 25 Sep 2020
It would probably be easier to get rid of out altogether and just sort the structure itself, e.g.:
files = files(idx(idy));
and then inside the loop you can simply access the folder and name, e.g.:
for k = 1:nume(files)
fnm = fullfile(files(k).folder,files(k).folder);
fid = fopen(fnm,'rt');
...
end

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!