Segmentation Faults when Running MEX files in Parallel
Show older comments
I am currently running repetitions of an experiment that uses MEX files in MATLAB 2012a and occasionally running into segmentation faults that I cannot understand.
Some information about the faults
- They occur randomly
- They only occur when I run multiple repetitions of my experiment in parallel on a Linux machine using a parfor loop.
- They do not occur when I run the repeitions of my experiment in parallel on my Mac using a parfor loop.
- They do not occur when I run or do they occur whenThey do not occur when I run the repetitions sequentially.
- They appear to occur far less frequently when I run 2 experiments in parallel - as opposed to 12 experiments in parallel.
- They occur in MATLAB 2011b and 2012a...
Some information about my MEX file:
- It is written in C
- It uses the IBM CPLEX 12.4 API (this is thread-safe)
- It was compiled using GCC 4.6.3
My thoughts are that there may be some kind of file access issue. Can anyone shed any light on what might be going on? I'd be happy to provide more information as necessary.
6 Comments
Jill Reese
on 17 Apr 2012
Could you post the code from the parfor loop? Is there a stack trace from the seg fault? What steps did you go through to compile the MEX file?
Berk Ustun
on 18 Apr 2012
Berk Ustun
on 18 Apr 2012
Edric Ellis
on 18 Apr 2012
Do you see the problem if you manually launch two MATLAB processes and run your MEX file? The workers that are running the body of your PARFOR loop are essentially ordinary MATLAB processes, but launched with the '-singleCompThread' option.
Berk Ustun
on 20 Apr 2012
jian shulei
on 8 Jan 2016
Edited: jian shulei
on 8 Jan 2016
I meet the same question,can you tell me how to do?
Answers (2)
James Tursa
on 17 Apr 2012
1 vote
Do you use any MATLAB API functions that allocate or change memory, such as mxCreateDoubleMatrix, mxMalloc, etc.? These are not thread-safe in general ... I don't know specifically about how parfor deals with this. What kind of file access are you doing?
Berk Ustun
on 17 Apr 2012
1 vote
8 Comments
James Tursa
on 17 Apr 2012
Whether malloc and free are thread-safe may depend on your implementation and compiler settings. I am not familiar with GCC.
Berk Ustun
on 18 Apr 2012
James Tursa
on 18 Apr 2012
Not necessarily. mxMalloc and mxFree *may* use malloc and friends in the background. But even if they do there is something else going on with mxMalloc and friends. MATLAB maintains a garbage collection list in the background of the addresses returned by mxMalloc & friends. The question is whether or not this garbage collection process is thread-safe or not. So the overall answer may not hinge 100% on just the thread-safeness of malloc etc.
Also, there seems to be plenty of evidence that the memory allocation/changing functions in the API are *not* thread-safe in an OpenMP environment. So the only way I can see them being thread-safe in a parfor situation is if parfor does something special in the background to interact with the mex API stuff to make it thread-safe. I really have no idea. I don't have the Parallel Toolbox to play with.
Walter Roberson
on 18 Apr 2012
No, mxMalloc and mxFree are not necessarily thread-safe even when malloc and free are. mxMalloc generally works with memory that already belongs to the process and could do its housekeeping in a way that is not thread safe. mxMalloc might call malloc() when it needs more memory, but the part before and the part after that might not be thread safe.
Berk Ustun
on 18 Apr 2012
James Tursa
on 18 Apr 2012
If you want the MATLAB memory manager to do garbage collection for you, then you need to use mxMalloc and mxFree. Otherwise you risk memory leaks for abnormal mex function exits unless you set up your own garbage collection scheme manually.
If you are creating MATLAB mxArray variables, officially speaking you must use the API functions. While it may be possible to create an mxArray variable from scratch using malloc and hacking into the mxArray structure itself, it would be of very limited use. E.g., you couldn't return such a variable back to a MATLAB workspace ... to do so would likely crash MATLAB when it tried to free this memory with its own memory manager when in fact the memory never came from this memory manager in the first place. There would also be a host of other problems as well that I won't go into here (variable sharing, etc). You really don't want to go down this path.
Since you have the Parallel Toolbox, I would check the doc to see if there is any mention of thread-safeness of the mex MATLAB API functions in a parfor setting. If not, maybe contact technical support to see if they can give you the answer.
James Tursa
on 18 Apr 2012
Another thought: Let's assume for the purposes of discussion that the mex API creation functions are not thread-safe. What you could do is create the "output" variables at the m-file level before calling the mex routine. Then pass these in as arguments and modify them in-place in the mex routine. That completely bypasses the thread-safeness issues of the mex API functions.
Berk Ustun
on 23 Apr 2012
Categories
Find more on Write C Functions Callable from MATLAB (MEX Files) in Help Center and File Exchange
on 17 Apr 2012
on 8 Jan 2016
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
