Constantly running out of memory - Would Parallel Computing fix this ?

hi,

the issue I'm having is not a new thing for the MatLab, the question is in regards to the parallel computing. Right now the computer specs are : 16GB ram Procesor Intel Xeon CPU E5-2676, 2.40 GHZ, 2400 Mhz. Can't even load the variables...

So is it Parallel computing a feasible solution ? or the local memory will always be the pain point ?

Oz

[Rishabh Rathore]

what are the sizes of the variables you are trying to load and are you getting any error messages?

[Oscar Omegna]

23 and 56gb , no just the run out of memory one

Answer 1

[Walter Roberson]

Parallel Computing always takes more memory as it has to duplicate objects.

Parallel Computing for memory management is only useful if you are using a cluster, in which case you might be distributed across nodes that do not share memory.

[Oscar Omegna]

I'm planning to do it in a cluster but loading the variables would still use local memory?

[Walter Roberson]

Are these large variables read-only for the computation? If so then you could look in the File Exchange for the Shared Matrix contribution. You might need one "real" copy of the data for each node, but with shared memory the individual CPUs would not need their own copy to just read the memory.

If you are also writing to these variables then you need to be careful about locking and race conditions.

If each worker only needs a well-defined slice of the data that the other workers do not need, then co-distributed arrays might help.

Answer 2

[John D'Errico]

Echoing what Walter has said - parallel processing is essentially never done to decrease memory load. If anything, it will more likely increase memory load.

Instead, you should consider using tools like tall arrays, tools that will help you to work with data not fully in RAM at once. Or use sparse matrices. There are many tools to decrease memory usage that have far more potential than anything you might get from parallel processing. Of course, you have not given any specifics of what you are doing, so it is difficult to know.