Matlab and parallel computing
3 views (last 30 days)
Show older comments
Hello,
I am facing some performance issues while reading data stored inside a wide list of txt files (around 1000000) and each file contains huge matrix data. I do not have PCT and I was thinking to use the java.util.concurrent package for multi-threading stuff (in this case I am looking to the BlockingQueue and ArrayBlockingQueue). Is there any expert that already used this package directly in a .m file? If yes, could you please share it? Do you see any limitation of MATLAB on using this package? Thank you in advance Best regards Paolo
1 Comment
Jan
on 11 Jul 2017
You have 1 million text files, which contain numerical data? This would be a really bad method to store the data, because many import and export methods work much more efficient with binary data.
Note that parallel processing does not help, when the single resource of the hard disk's hardware is the bottleneck.
Accepted Answer
John D'Errico
on 11 Jul 2017
As Jan says, you are probably wasting your time trying to use parallel processing to solve a problem that is not CPU bound, but disk bound.
I'd suggest a better tool is a RAM drive. Speed up the access time instead.
And having them as text files just means that your read time will be terribly slow. That may be something beyond your control.
Finally, I'm not sure this is a good task for MATLAB to solve. If you need the information in MATLAB eventually, then collect it all using some other tool, then read the entire mess to MATLAB in one piece.
0 Comments
More Answers (0)
See Also
Categories
Find more on Startup and Shutdown in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!