Matlab and parallel computing
4 views (last 30 days)
I am facing some performance issues while reading data stored inside a wide list of txt files (around 1000000) and each file contains huge matrix data. I do not have PCT and I was thinking to use the java.util.concurrent package for multi-threading stuff (in this case I am looking to the BlockingQueue and ArrayBlockingQueue). Is there any expert that already used this package directly in a .m file? If yes, could you please share it? Do you see any limitation of MATLAB on using this package? Thank you in advance Best regards Paolo
John D'Errico on 11 Jul 2017
As Jan says, you are probably wasting your time trying to use parallel processing to solve a problem that is not CPU bound, but disk bound.
I'd suggest a better tool is a RAM drive. Speed up the access time instead.
And having them as text files just means that your read time will be terribly slow. That may be something beyond your control.
Finally, I'm not sure this is a good task for MATLAB to solve. If you need the information in MATLAB eventually, then collect it all using some other tool, then read the entire mess to MATLAB in one piece.