reading multiple text file from two different folders using nested for loop

i am using the following lines to read multiple text files and do some comparison between them. Baiscally the first loop will read text files of certain format (we have 30 files of this type). the second loop will read text files of another format (we have 33000 of them) and we do some comparison between them.

I have my code working and it is fine but it is extremely slow because of the nested loop, and its important to mention that i dont really need these loop for anything else, only for reading the files from folders.

So is there a more effecient way of doing this.

for n = 1 :nfiles2
     fprintf('.......... File - %d of %d\n',n, nfiles2)
     fullFileName2 = fullnames2{n};
     t2 = readtable(fullFileName2);  %t2 stores the ionosonde profiles
     

     for m = 1 :10
       fprintf('.......... insideFile - %d of %d\n',m, 10)
     fullFileName1 = fullnames1{m};
     t1 = readtable(fullFileName1); %t1 stores the ISR profiles

Answer 1

[Konrad]

Hi Salma

It depends on what exactly you want to do with the data in the text files, but reading the file line by line seems to be considerably faster compared to readtable()

tic;
for k = 1:100
    t = readtable('test.csv');
end
toc

Elapsed time is 1.449934 seconds.

tic;
for k = 1:100
    fid = fopen('test.csv');
    clear tl;
    tl{1} = fgetl(fid);
    while ischar(tl{end})
        tl{end+1} = fgetl(fid);
    end
    fclose(fid);
end
toc

Elapsed time is 0.633903 seconds.

Of course readtable() converts numbers to double while fgetl() returns char and if you have to do type conversion the speed gain might disappear. But I think its worth trying.

Best, Konrad

[Salma fathi]

thank you for your reply, I have tried what you suggested, it is much more faster than the way I used to do it.

but I get the content of the textfiles in the following form, each row of the file is stored in a cell

while we want it to be in the following form

Would there be a way to do that?

[Konrad]

Yes, thats what I ment with type-conversion ;)

I assume your data is comma delimited.

get the header in the first row:

fid = fopen('test.csv');
tline = fgetl(fid);
header = strsplit(tline,','); % <- the variable names as cellstring

If the data is all numeric (with variable names in the first row) you can do the following:

tic;
data = csvread('test.csv',1,0);
toc

Elapsed time is 4.558796 seconds.

Else: you can read the text data, store it in a cellstring and convert the numeric part (you will have to know which columns you need). This takes much longer due to the type conversion (and the growing array, which could be avoided by preallocating), but could be made faster if you only need a small fraction of the data for your comparison. But I don't know what you want to compare, so here's how you get everything from the test.csv:

tic;
dataCellstr = cell(0);
selectedCols = 1:3;
tline = fgetl(fid);
while ischar(tline)
    cols = strsplit(tline,',');
    dataCellstr(end+1,:) = cols(selectedCols);
    tline = fgetl(fid);
end
data2 = str2double(dataCellstr);
fclose(fid);
toc

Elapsed time is 15.281463 seconds.