Use datastores and tall arrays.
Python-like generator for reading large CSV file?
6 views (last 30 days)
Show older comments
I have a large csv file, typically I just read everything into a array like this
num =csvread(filename,2,2)
if csv file is big, the num array will be big as well and sometime it blowout my memory.
In python, there is a way to read the file but without holding the data in the memory. the methods use generator object. For example,
Base case(w/o using generator object):
value=[len(x) for x in open('filename.tx')] this case will hold the data in the memory
another case(use generator object:
it=(len(x) for x in open('filename.tx'))
the it is a generator object and it only hold the memory size for the generator expression without holding memory for all the data, later, we just extract the data by using next(it).
I am wondering if Matlab has similar generator object as Python when reading csv file to save the memory? Thanks
0 Comments
Accepted Answer
More Answers (1)
See Also
Categories
Find more on Standard File Formats in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!