Using importdata to import sorted files

I have a file folder that contains about 77,000 matlab files. I need to read in these files, and then in a loop, extract one value from one pixel of each (representing temperature at the selected location) to create a times series for that location. For example, I would create an array that contained the temperature for location (260,80) from all files.

The files are named "T_num2str(i)_num2str(k).mat". The first string of numbers remains the same for 1632 files, then adds one digit, stays the same for 1632 files, then adds one, etc. So for example the files are T_1160_1.mat through T_1160_1632.mat, T_1161_1.mat through T_1161_1632.mat, etc. I first tried to just copy and rename the filenames to be sequential numbers without characters, but the script was slated to take 76 days to run...I cannot rename the files without copying them, so ideally I can just use a script that will read the existing file names.

My code will successfully do this for one file, but once in a loop I am having issues with importdata saying it cannot find the file. My guess is that I am not correctly scripting the filenames. Can someone look at this code and suggest why my files may not be importing?

Note that I used the natsortfiles function because when matlab is reading the files, otherwise it is sorting:

T_1640_1.mat

T_1640_10.mat

T_1640_2.mat

Natsortfiles creates a cell of the file names. I tried using cell2mat thinking that may be the importdata issue but I was not successful. I have uploaded an example file (T_1161_1.mat)

new_path_dir='Z:\myfolder\test\'; %file directory location
filelist = dir(fullfile(new_path_dir,'*.mat')); % get list of files in directory
sorted = natsortfiles({filelist.name}); % sort file names into order
for i=1160:1208
    for k=1:numel(sorted)   
        temp=strcat(new_path_dir,'T_',num2str(i),'_',num2str(k),'.mat');
        temp2=importdata(temp);
        value = getfield(temp2,strcat('T_',num2str(i),'_',num2str(k)));
        extracted(1,k)=value(260,80);
    end
end

Answer 1

[Stephen23]

Your code is a bit mixed up, because you are confusing two different paradigms for reading multiple files:

• reading the filenames from the OS (e.g. using dir).
• generating the filenames from a variable (e.g. loop iterator).

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

Both of these might be acceptable methods for reading your files, but it makes little sense to try and combine them together. You should pick one, and stick with that. Here is a simple example of using dir, which should work for your files:

P = '.'; % directory where the files are.
S = dir(fullfile(P,'*.mat'));
S = natsortfiles(S); % alphanumeric sort by filename
for k = 1:numel(S)
    T = load(fullfile(P,S(k).name));
    [~,F] = fileparts(S(k).name);
    S(k).data = T.(F);
end

Note how loading the field would be simpler if it had the same name in each file, because then all that would be needed would be:

S(k).data = T.fieldname;

I tested my code on the file that you uploaded and it worked as expected.

[geog170]

Thank you Stephen for the advice! I just started using Matlab a few months ago so I am absolutely sure that my code is inefficient, at best. I can follow everything that you have done to read in the files, and it is working well on my test set. I can also see where the temperature values are stored in C{1,1}. In the for loop I have tried adding back in my line for extracting the pixel value at (280,60) for example, but I am receiving the error "Index exceeds matrix dimensions." Should this line be modified since C is a cell now?

[Stephen23]

@wxstudent25: it is possible to put the pixel indexing after the field, like this:

T.(F)(260,80)

and everything else is the same. But if you really only need one pixel from each iteration then using a cell array is overkill and just complicates things, so you could use a numeric array, like this:

M = nan(size(N)); % preallocate!
for k = ...
    ...
    M(k) = T.(F)(260,80);
end

If that looks awkward to you, remember that you can always use temporary variables to store data:

tmp = T.(F);
M(k) = tmp(260,80);

[geog170]

This has been very informative and helpful, thank you!