Shared Memory in parfor - Parallel Computing Toolbox - sharedmatrix does not work in 2016b

Hi,

I'm trying to optimize a simulation that uses a large array in a parfor loop which is used as read-only.

The data is copied to each worker although it is not changed by the workers. This is a huge waste of memory...

There is a nice mex function wrapping shared memory from Boost: https://de.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix

However, this does not work anymore in Matlab 2016b.

Is there another possibility to share memory between workers? Or is there an easy fix to the sharedmatrix mex function?

(The problem with sharedmatrix seems to be that 'detach' does not work anymore and MATLAB crashes when attempting to delete the temporary pointer to the shared memory)

Thanks,

Matthias

Answer 1

[Harsh]

Hello,

Parallel Computing Toolbox” follows a distributed memory model. Hence the workers won’t have access to the data present on the client machine. Instead, the client copies the data, in this case the large array to each worker when you use the “parfor” construct.

In the event that the workers are each working on different chunks of the large array, you can potentially look at using (co)distributed arrays as shown in the following links:

https://www.mathworks.com/help/distcomp/distribute-arrays.html

http://www.mathworks.com/help/distcomp/working-with-codistributed-arrays.html

https://www.mathworks.com/help/distcomp/choosing-a-parallel-computing-solution.html

[Joao Henriques]

This memory model makes sense for remote clusters, but not for local workers, which is where the described problem arises.

As the original poster noted, this is extremely wasteful for read-only arrays, easily cutting the amount of memory you can process by a large factor.

[Florian Enner]

You should be able to use 'memmapfile' for this

[Michael Jablecki]

I agree ... that works.