You are now following this question
- You will see updates in your followed content feed.
- You may receive emails, depending on your communication preferences.
batch processing (automating code processing)
5 views (last 30 days)
Show older comments
Hi all,
I have a question about running a code on multiple datasets. I have seen many examples and methods, but unfortunetly, they did not work for me.
I have over 500 data sets, all have same structure of data but different numbers (arranged in sruct.array) so they are already in .mat form.
Those .mat files are stored on my computer as follow :mainfolder > 5 subfolder > N number of subsubfolder > 15 .mat files
Thus, for each subsusbfolder, there is 15 .mat files.
So, basically I want to process those .mat files by calling a function on all of those data sets automatically, but not really sure how to do so especially that I need a for loop to navigate my subfolders and subsubfolders to reach to the .mat files.
Any help would be appreciated.
Accepted Answer
Mario Malic
on 2 Nov 2020
Edited: Mario Malic
on 2 Nov 2020
This will give you the structure containing all .mat files.
File_Struct = dir('**/*.mat');
Edit:
Use name and folder to get the full path to each .mat file.
File_Struct = dir(fullfile(myFolder, '**/*.mat'))
File_Path = cell(size(File_Struct,1), 1);
for ii = 1 : 1 : size (File_Struct,1)
File_Path {ii} = fullfile(File_Struct(ii).folder, File_Struct(ii).name);
fprintf(1, 'Now reading %s\n', File_Path {ii});
load(File_Path {ii}); %this is changed
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
Processed_Data(ii).Value = data; % structure
end
23 Comments
maryam al labbad
on 2 Nov 2020
Edited: maryam al labbad
on 2 Nov 2020
Thanks for the quick response. this piece of code load all .mat files into matlab, but I am not sure how to run the 2 functions I have to each .mat file.
Steven Lord
on 2 Nov 2020
This does not load anything into MATLAB. It gives you a list of file information through which you can iterate (with a for loop) to load and process.
cd(matlabroot)
cd toolbox/matlab
theFiles = dir('**/*.mat');
In my installation of release R2019b, one of the files in the demos directory is element 40 of theFiles.
>> theFiles(40)
ans =
struct with fields:
name: 'trimesh3d.mat'
folder: 'C:\Program Files\MATLAB\R2019b\toolbox\matlab\demos'
date: '25-Nov-2008 12:44:41'
bytes: 24987
isdir: 0
datenum: 733737.531030093
To load this file:
>> F = theFiles(40);
>> fn = fullfile(F.folder, F.name)
fn =
'C:\Program Files\MATLAB\R2019b\toolbox\matlab\demos\trimesh3d.mat'
>> data = load(fn)
data =
struct with fields:
tri: [2174×3 double]
x: [1079×1 double]
y: [1079×1 double]
z: [1079×1 double]
You can do whatever is appropriate with the data from the file. In this case it's a meshed object so trimesh is appropriate:
>> trimesh(data.tri, data.x, data.y, data.z)
maryam al labbad
on 2 Nov 2020
Thanks Steven, I am a beginner on matlab, so my apologies if I am asking basic questions.
Do I have to indicate the main folder directory? like I am not sure from where to start. I have this piece of code that I am trying to let it work, but it says that unrecognize function or variable data although when I am calling these 2 functions outside of the loop on each .mat file, they work fine. (each .mat file is named data.mat and it has the inputs for each function)
myFolder = 'C:\Users\m\Desktop\try'; % main folder
if ~isdir(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
uiwait(warndlg(errorMessage));
return;
end
filePattern = fullfile(myFolder, '*.mat');
matFiles = dir(filePattern);
for k = 1:length(matFiles)
baseFileName = matFiles(k).name;
fullFileName = fullfile(myFolder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
matData(k) = load(fullFileName);
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
end
maryam al labbad
on 2 Nov 2020
Mario, I really appreciate your efforts in helping me out, do I have to include main folder directory, so the code should be
myFolder = 'C:\Users\m\Desktop\try'; % main folder
File_Struct = dir('**/*.mat');
File_Path = cell(size(File_Struct,1), 1);
for ii = 1 : 1 : size (File_Struct,1)
File_Path {ii} = fullfile(File_Struct(ii).folder, File_Struct(ii).name);
fprintf(1, 'Now reading %s\n', File_Path {ii});
matData(ii) = load(fullFileName); % matData is not used in the code here?
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
end
Mario Malic
on 2 Nov 2020
This works if you would like to look for files within other folders.
File_Struct = dir(fullfile(myFolder, '**/*.mat'))
Without specifying the folder, it will look for all .mat files within current folder and subfolders.
maryam al labbad
on 2 Nov 2020
File_Struct = dir(fullfile(myFolder, '**/*.mat'))
File_Path = cell(size(File_Struct,1), 1);
for ii = 1 : 1 : size (File_Struct,1)
File_Path {ii} = fullfile(File_Struct(ii).folder, File_Struct(ii).name);
fprintf(1, 'Now reading %s\n', File_Path {ii});
matData(ii) = load(fullFileName); % matData is not used in the code here?
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
end
Hi Mario,
With the above code, I am not getting anything. Am I missing a core piece?
File_Struct =
0×1 empty struct array with fields:
name
folder
date
bytes
isdir
datenum
maryam al labbad
on 2 Nov 2020
Edited: maryam al labbad
on 2 Nov 2020
I may not explain my question clearly, here is a summary of how my files look like. I want to run same 2 fucntions on each .mat file.
C:\Users\m\Desktop\try
+---subfolder_1
+-----data1.mat
+-----data2.mat
+-----data3.mat
+-----data4.mat
+-----data5.mat
+-----data6.mat
+-----data7.mat
+-----data8.mat
+-----data9.mat
+-----data10.mat
+-----data11.mat
+-----data12.mat
+-----data13.mat
+-----data14.mat
+-----data15.mat
+---subfolder_2
+-----data1.mat
+-----data2.mat
+-----data3.mat
+-----data4.mat
+-----data5.mat
+-----data6.mat
+-----data7.mat
+-----data8.mat
+-----data9.mat
+-----data10.mat
+-----data11.mat
+-----data12.mat
+-----data13.mat
+-----data14.mat
+-----data15.mat
+---subfolder_3
+-----data1.mat
+-----data2.mat
+-----data3.mat
+-----data4.mat
+-----data5.mat
+-----data6.mat
+-----data7.mat
+-----data8.mat
+-----data9.mat
+-----data10.mat
+-----data11.mat
+-----data12.mat
+-----data13.mat
+-----data14.mat
+-----data15.mat
+---subfolder_4
+-----data1.mat
+-----data2.mat
+-----data3.mat
+-----data4.mat
+-----data5.mat
+-----data6.mat
+-----data7.mat
+-----data8.mat
+-----data9.mat
+-----data10.mat
+-----data11.mat
+-----data12.mat
+-----data13.mat
+-----data14.mat
+-----data15.mat
+---subfolder_5
+-----data1.mat
+-----data2.mat
+-----data3.mat
+-----data4.mat
+-----data5.mat
+-----data6.mat
+-----data7.mat
+-----data8.mat
+-----data9.mat
+-----data10.mat
+-----data11.mat
+-----data12.mat
+-----data13.mat
+-----data14.mat
+-----data15.mat
Mario Malic
on 2 Nov 2020
Edited: Mario Malic
on 2 Nov 2020
File_Struct can't be empty, commands above will return .mat files frorm folder C:\Users\m\Desktop\try and all subfolders within it. Do you have a typo in your folder, or files are not there?
maryam al labbad
on 2 Nov 2020
Edited: maryam al labbad
on 2 Nov 2020
With the following code, I can access all .mat files in the folder.
% Specify the folder where the files live.
myFolder = 'C:\Users\m\Desktop\try\WSW';
% Check to make sure that folder actually exists. Warn user if it doesn't.
if ~isfolder(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
uiwait(warndlg(errorMessage));
myFolder = uigetdir(); % Ask for a new one.
if myFolder == 0
% User clicked Cancel
return;
end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '**/*.mat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
baseFileName = theFiles(k).name;
fullFileName = fullfile(theFiles(k).folder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
end
However, I cannot call the functions. It shows the following error message:
Unrecognized function or variable 'data'.
Error in Untitled14 (line 20)
data=myfunc1(data);
Mario Malic
on 2 Nov 2020
Edited: Mario Malic
on 2 Nov 2020
I can't see what file contains as I do not have the file. Actually, I don't know how your code works, data is not defined in it. I did write this on my answer above.
matData(ii) = load(fullFileName); % matData is not used in the code here?
Documentation for the load function:
- If filename is a MAT-file, then S is a structure array.
matData(ii) = load(fullFileName); % matData is not used in the code
% data = load(fullFileName); % I can't tell is this correct
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
You have to fix these lines according to the documentation above, I don't know what these functions do and is data overwritten or appended, therefore I can't really know how to proceed with this.
maryam al labbad
on 2 Nov 2020
Edited: maryam al labbad
on 2 Nov 2020
Sorry for making all the confusions. I hope I can make it clear in this post. The last code I posted is different than the first one. The last one showed me all files on my folders and subfolders in matlab, the previous one was showing zero if you recall. So the code that I am working on right now is:
% Specify the folder where the files live.
myFolder = 'C:\Users\m\Desktop\try\WSW';
% Check to make sure that folder actually exists. Warn user if it doesn't.
if ~isfolder(myFolder)
errorMessage = sprintf('Error: The following folder does not exist:\n%s\nPlease specify a new folder.', myFolder);
uiwait(warndlg(errorMessage));
myFolder = uigetdir(); % Ask for a new one.
if myFolder == 0
% User clicked Cancel
return;
end
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '**/*.mat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
baseFileName = theFiles(k).name;
fullFileName = fullfile(theFiles(k).folder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
data=myfunc1(data)% calling my function1
data=myfunc2(data)% calling my function2
end
Each .mat file has two type of data stored in structure array such as the following:
data.probe(1).U
data.probe(1).Y
data.probe(1).P
data.probe(2).U
data.probe(2).Y
data.probe(2).P
so my functions :
data=myfunc1(data)
data=myfunc2(data)
are used to compute some statsitics and calculate some paramters based on U, Y, and P. both functions are exactly the same, the only difference is that 1st function will compute data and store it in(1), and 2nd function will compute the data for (2).
The .mat file name is data which has U,Y, and P, which are the input for the functions to calculate rest of parameters. Thus, when I run my functions, it usually compute the rest of parameters on the same .mat file but different cells.
Mario Malic
on 2 Nov 2020
Edited: Mario Malic
on 2 Nov 2020
I have edited the code in the answer, please adjust variable names to your liking. Below code is not well understood, these asterisks are called wildcards, where any files with .mat extension will be added to filePattern. This example below is not very intuitive because it also searches in subfolders.
filePattern = fullfile(myFolder, '**/*.mat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
Consider this example
filePattern = dir(fullfile(myFolder, '*.mat'));
This will get you .mat files only in myFolder.
maryam al labbad
on 2 Nov 2020
Thanks Mario, this seems to be working! but the thing is I think all .mat files are overwritten each another. What I want to do is to save the file after each iteration because this way, I am not able to get the data except for the last one. Please guide me, I really apprecaite your help
Mario Malic
on 2 Nov 2020
I have edited the answer, I don't know what the outputs are so, data for each iteration is saved in structure Processed_Data under field Value.
maryam al labbad
on 2 Nov 2020
Mario, that works perfect, have been trying to resolve this for few days, so many thanks, I really apprecaite your time
Mario Malic
on 3 Nov 2020
You're welcome, for the future questions, you can read the help and a tutorial for the questions so that they are faster resolved. Have a great rest of the week!
maryam al labbad
on 3 Nov 2020
Hello again, I am wondering if there is a way to save those files seperatly in a specific directory? now I am getiing all of them in a structure, just wondering how to save them seperatly? Thanks a lot
Mario Malic
on 3 Nov 2020
Edited: Mario Malic
on 3 Nov 2020
You can use function save, you can check it in the documentation.
You can put this line of code in the loop on the appropriate place in the loop. I am not familiar if you're having one or more files in a single folder. You will have to modify the name between each file so it doesn't get overwritten.
save(fullfile(File_Struct(ii).folder, File_Struct(ii).name), 'data');
I did not test this, this will overwrite each .mat file in folder.
However, I would suggest not do it this way, because it will take long to load and save, especially if you have a lot of files. You can add additional identifier in the structure for each analysed file/folder.
maryam al labbad
on 3 Nov 2020
Edited: maryam al labbad
on 3 Nov 2020
Hi, let's say I want to run more than one function on the code, and I want to save the result of other function. I did that, but it overwrite the saved file. I changed data in the following code to the other function name, but it seems that it overwite the data file in the folder. How can I make it save a new file of the result of the other functions. does it deal with .name thing?
Thanks
save(fullfile(File_Struct(ii).folder, File_Struct(ii).name), 'data');
save(fullfile(File_Struct(ii).folder, File_Struct(ii).name), 'data2');
Mario Malic
on 3 Nov 2020
This time, I'll suggest you to read the documentation on the functions used in these two lines. I want to help, but if I'm going to answer every question you might have, it won't lead you further ahead.
So, read up on save and fullfile.
What value does fullfile return? What input arguments save function takes? Once you learn that, you'll be able to solve your problem. You were on the right track with the answer - does it deal with .name thing?
More Answers (0)
See Also
Categories
Find more on File Operations in Help Center and File Exchange
Tags
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!An Error Occurred
Unable to complete the action because of changes made to the page. Reload the page to see its updated state.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
Asia Pacific
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)