Does the MATLAB python interface support parallel processing?

I am trying to access a simple python function via Matlab in R2022a:

function out = lognormcdf (x)

out = double(py.scipy.stats.norm().logcdf(x));

end

When I call call this function repeatedly in a for loop, the output is as expected, but in a parfor loop, I get the error attatched at the end of this file. I guess that every worker is accessing the same python session, and are interfering with one another. Is there any solution to this?

This is my error trace:

Starting parallel pool (parpool) using the 'local' profile ...

Connected to the parallel pool (number of workers: 10).

Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again on the remaining workers.

> In distcomp/remoteparfor/handleIntervalErrorResult (line 245)

In distcomp/remoteparfor/getCompleteIntervals (line 395)

In parallel_function>distributed_execution (line 746)

In parallel_function (line 578)

In plot_mle_vs_d (line 26)

Error using distcomp.remoteparfor/rebuildParforController

All workers aborted during execution of the parfor loop.

Error in distcomp.remoteparfor/handleIntervalErrorResult (line 258)

obj.rebuildParforController();

Error in distcomp.remoteparfor/getCompleteIntervals (line 395)

[r, err] = obj.handleIntervalErrorResult(r);

Error in plot_mle_vs_d (line 26)

parfor i = 1:n_mc

Preserving jobs with IDs: 13 because they contain crash dump files.

You can use 'delete(myCluster.Jobs)' to remove all jobs created with profile local. To create 'myCluster' use 'myCluster = parcluster('local')'.

The client lost connection to worker 7. This might be due to network problems, or the interactive communicating job might have

errored.

Warning: 10 worker(s) crashed while executing code in the current parallel pool. MATLAB may attempt to run the code again on the

remaining workers of the pool, unless an spmd block has run. View the crash dump files to determine what caused the workers to crash.

Answer 1

[Raymond Norris]

I can't see a single python session being the issue. I'm betting you ran out of memory. Try the following

• I'm assuming you have (at least) 10 cores. Start a smaller number of workers.
• How much memory do you have? Aim for about 4 GB/per worker. How big is x? The reason this is less of a concern serially is that you only have 1 MATLAB running (versus 10).
• Create the python session within in the parfor (to remove this concern)

[Justin Kang]

I think you are incorrect:

(1) I have enough cores

(2) Memory is not a concern, my task requires very little, and when I replace the function with a native matlab one, there is no issue. Unless the python interface causes a 100x or 1000x increase in memory, this is not the issue.

(3) I don't think MATLAB can do this. If I try I get this:

Error using plot_mle_vs_d

Python is loaded. The environment cannot be changed in this MATLAB session. To change the environment, restart MATLAB, and then call 'pyenv'.

(4) This is a error report:

--------------------------------------------------------------------------------

Segmentation violation detected at 2023-05-16 23:24:31 -0700

--------------------------------------------------------------------------------

Configuration:

Crash Decoding : Disabled - No sandbox or build area path

Crash Mode : continue (default)

Default Encoding : UTF-8

Deployed : false

Graphics Driver : Uninitialized software

Java Version : Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode

MATLAB Architecture : maci64

MATLAB Entitlement ID : 4808126

MATLAB Root : /Applications/MATLAB_R2022a.app

MATLAB Version : 9.12.0.2039608 (R2022a) Update 5

OpenGL : software

Operating System : Mac OS Version 13.0.1 (Build 22A400)

Process ID : 16297

Processor ID : x86 Family 6 Model 44 Stepping 0, GenuineIntel

Session Key : d7972c12-9cb1-41bb-b7c2-73f7cc80e3ad

Window System : None

Fault Count: 1

Abnormal termination:

Segmentation violation

Current Thread: 'MCR 0 interpreter thread' id 0x307118000

Register State (from fault):

RAX = 0000000000000001 RBX = 000000030710ffd0

RCX = 0000600004c6d0a0 RDX = 0000000300000001

RSP = 000000030710f278 RBP = 000000030710fdb0

RSI = 0000600004c6d0a0 RDI = 0000000000000001

R8 = 0000000000000002 R9 = 0000600000000002

R10 = 0000000300000001 R11 = ffff9ffcffffffff

R12 = 0000600004c6d0a0 R13 = 000000030710ffd8

R14 = 0000600004c6d0a0 R15 = 000000030710ffe0

RIP = 000000016759a1ad RFL = 0000000000000297

CS = 000000000000002b FS = 0000000000000000 GS = 0000000000000000

Stack Trace (from fault):

[ 0] 0x000000010b849894 /Applications/MATLAB_R2022a.app/bin/maci64/libmwfl.dylib+00325780 _ZN10foundation4core4diag15stacktrace_base7captureERKNS1_14thread_contextEm+00000052

[ 1] 0x000000010b80134a /Applications/MATLAB_R2022a.app/bin/maci64/libmwfl.dylib+00029514 _ZN10foundation4core4test17terminate_handledERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE+00006986

[ 2] 0x000000010b7ff505 /Applications/MATLAB_R2022a.app/bin/maci64/libmwfl.dylib+00021765 _ZN10foundation4core4diag13terminate_logEPKcPK17__darwin_ucontext+00000149

[ 3] 0x0000000113f266bb /Applications/MATLAB_R2022a.app/bin/maci64/libmwmcr.dylib+00526011 _Z19mnPrintErrorMessageRKNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEE+00012203

[ 4] 0x0000000113f24662 /Applications/MATLAB_R2022a.app/bin/maci64/libmwmcr.dylib+00517730 _Z19mnPrintErrorMessageRKNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEE+00003922

[ 5] 0x0000000113f21ee5 /Applications/MATLAB_R2022a.app/bin/maci64/libmwmcr.dylib+00507621 mnFatalSignalHandler+00000149

[ 6] 0x00007ff807371c1d /usr/lib/system/libsystem_platform.dylib+00015389 _sigtramp+00000029

[ 7] 0x0000000162d7eb04 <unknown-module>+00000000

[ 8] 0x00000001666f9e59 /Applications/MATLAB_R2022a.app/bin/maci64/mkl.dylib+00216665 mkl_blas__sdot+00000153

[Justin Kang]

@Raymond Norris Shall I report this as a bug?

[Raymond Norris]

Please do, @Justin Kang. Thanks.

[Ed Wheatcroft]

@Justin Kang did anything ever come out of this? I'm having a similar issue