Using readtable (or similar) to import multiple ranges of data

Hi team,

I need to import some data from spreadsheets (.csv), my issue is, is thast the data is scattered throughout a single column vector, I have the row indexes for the data I need, but I am not sure how to go about implementing this using readtable or similar.

I have about 117,000 .csv files that i need data from, the data is all in the same place, so a simple for loop would do this, I have attempted to do this, but the time taken for this to process is extremely large ( I understand I have a lot of .csv files to go through).

is there anyway to increase the finishing time i.e., using an alternative to a for loop?

% range = 35 values not in sequential order
for i=3:(height(filedir))
    file = convertCharsToStrings(filedir(i).name);
    S{i-2} = readtable(file,'Range',range,'ReadVariableNames',false);
end

SCADA = [];
for i = 1:width(S)
    arr = table2array(S{i});
    SCADA = [SCADA; arr];
end
save SCADA.mat SCADA

this current form of the code imports ALL data in the column, some 300 odd ..

any help would be amazing!

Answer 1

[J. Alex Lee]

it's hard to say without seeing the csv file, but it sounds like the data is not really tabular, and you may be better off using "readmatrix" or if there's too much overhead, some lower level function. See if below works

% probably best to use readmatrix and avoid your post processing
filedir = dir("folder/*.csv"); % use filter to only return .csv files

% then you don't have to play games with indexing
for i = numel(filedir):-1:1
    file = convertCharsToStrings(filedir(i).name);
    S{i} = readmatrix(file,'Range',range,'ReadVariableNames',false);
end
SCADA = vertcat(S{:});

By the way, another thing that may be taking a lot of time is growing your array SCADA continually (especially 117,000 times)

Try either of these to see if it brings down your time significantly if you want to stick with readtable

% intuitively to me, this should be faster
T = vertcat(S{:});
SCADA = table2array(T);

or

for i = numel(S):-1:
    M{i} = table2array(S{i});
end
SCADA = vertcat(M{:});