Stacking more than 200K files (same #columns)

3 views (last 30 days)
Dave
Dave on 13 Oct 2021
Commented: Dave on 13 Oct 2021
Hello, I have more than 200,000 csv files I would like to put/stack/append/concatenate over each other.
All csv file are in the same folder.
All csv files have the same number of columns (80)
But they may have different number of rows (from 5 to 5000)
I am currently using a loop and readtable for each file
out1=readtable(csvfilename{ii},opts);
out2=[out2;out1];
But it has taken forever, and likely out2 may be too big. Is this the correct way to deal with 200k files? Tall tables?
From the 80 columns in the csv, I only need the same 30 columns, this in case I can directly read those 30 to avoid making the final file too big.

Answers (1)

Cris LaPierre
Cris LaPierre on 13 Oct 2021
Edited: Cris LaPierre on 13 Oct 2021
Consider using a datastore. You can see an example in our Importing Multiple Data Files video from our Practical Data Science with MATLAB specialization on Coursera.
I'm not sure what 'taking forever' means, but it is going to take a while to load 200k files. because the array size hasnt' been preallocated, you are going to encounter memory issues as the array grows, as MATLAB has to keep moving it to larger and larger blocks of continuous membory (see here) If the size gets too big, you may need to look into using a tall array in order to work with the final result. In that case, you may want to look into a TallDatastore.
  4 Comments
Dave
Dave on 13 Oct 2021
Thanks, to avoid the error I am using
ds1=datastore('C:...*.csv')
ds1.TextscanFormats{9}='%q'
ds2=readall(ds1);
I'm trying in a smaller set of 5 files (vs 200k) and it works
But, is there a way to use readall (or read) to get specific columns only?
specCol={'var1', 'var9','var65'}
However running:
ds1.SelectedVariableNames=specCol;
I get
Error using matlab.io.datastore.TabularTextDatastore/set.SelectedVariableNames (line 714)
SelectedVariableNames must be a unique subset of VariableNames.
Dave
Dave on 13 Oct 2021
nvm, there was a typo in a variable name.

Sign in to comment.

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!