Why OpenMP does not work in a CUDA code compiled in Matlab (as a MEX)？

Hi, I'm trying to use OpenMP in my CUDA code, compile and use it in Matlab. However, it does not work. Here is a simple example:

#include <cuda_runtime.h>
#include "device_launch_parameters.h"
#include <stdio.h>
#include "cuda.h"
#include "mex.h"
#include "omp.h"

void mexFunction(int nlhs, mxArray* plhs[],
	int nrhs, const mxArray* prhs[])
{

#pragma omp parallel
	{
		int ID = omp_get_thread_num();
		printf("Hello(%d)", ID);
		printf("World(%d) \n", ID);
	}

}

I use

mexcuda OpenMPTest.cu 

to compile this code in Matlab. When i run this in Matlab, all I get is "Hello(0)World(0) ". So, something is wrong. Maybe the probem is with the way that I compile?!!! Please help.

Moein.

[Walter Roberson]

See also https://www.mathworks.com/matlabcentral/answers/1448809-how-can-i-compile-a-mex-function-which-contain-openmp-and-cuda#comment_1739449

[Moein Mozaffarzadeh]

Hi Walter,

I have already seen that post. It does not provide any compiling solution.

could you please let me know how i can add liomp5 to my build line?

Searching through the web, i got this one so far, but no changes! The code gets compiled well, but ThreadID is always zero.

mexcuda C:\ProgramData\MATLAB\SupportPackages\R2021a\3P.instrset\mingw_w64.instrset\lib\gcc\x86_64-w64-mingw32\6.3.0\libgomp.a -v CFLAGS="$CFLAGS -fopenmp" -LDFLAGS="$LDFLAGS -fopenmp" Xcompiler="-fopenmp" OpenMPTest.cu

[Walter Roberson]

-liomp5 is probably the flag needed; you might also need a -L flag to point to the directory the library is in.

The -l part is a flag, and iomp5 is its argument, but by convention the linker will automatically try prefixing it with l and lib to search for it. You can also give a specific name and extension

I did not realize that OpenMP could be used with CUDA, so this is not something I have experience with myself.

[Joss Knight]

Sorry, I'm not an OpenMP expert. Are you sure you've specified that more than one thread should be created? Don't you have to compile with -fopenmp, which you might need to add to the CXXFLAGS variables when invoked using MEX?

[Walter Roberson]

Ah, add the -fopenmp to the CXXFLAGs instead of the current CFLAGS location ?

[Moein Mozaffarzadeh]

I added -fopenmp to the CXXFLAGs instead of the current CFLAGS, but still the threadID is zero.

I cannot find iomp5 anywhere. Any idea at what directory this could be found?

[Moein Mozaffarzadeh]

I just tried the following .cpp code to see if the problem is with my OMP or the cuda compiler is the problem.

#include <stdio.h>
#include "mex.h"
#include "omp.h"

void mexFunction(int nlhs, mxArray* plhs[],
	int nrhs, const mxArray* prhs[])
{
	omp_set_num_threads(4);
#pragma omp parallel
	{
		printf("Hello(%d), ", omp_get_thread_num());
		printf("World(%d) \n", omp_get_thread_num());
	}
}

I used the following code

mex C:\ProgramData\MATLAB\SupportPackages\R2021a\3P.instrset\mingw_w64.instrset\lib\gcc\x86_64-w64-mingw32\6.3.0\libgomp.a -v ...
      CXXFLAGS="$CXXFLAGS -fopenmp" LDFLAGS="$LDFLAGS -fopenmp" Xcompiler="-fopenmp"  OpenMPTest.cpp

to compile. Unfortunately, it does not even compile. I get the following error:

... OpenMPTest.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in

Any idea what is the problem now?

[Walter Roberson]

https://stackoverflow.com/questions/11079586/getting-started-with-openmp-install-on-windows

https://software.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/optimization-and-programming-guide/openmp-support/openmp-library-support/openmp-support-libraries.html (if you were using Visual Studios)

[Bruno Luong]

OpenMP under windows compilers are (from this fileexchange)

   cc = cc(loc);
    if contains(cc.ShortName,'MSVC')
        opmopt = {'COMPFLAGS="$COMPFLAGS /openmp"'};
        
    elseif contains(cc.ShortName,'INTEL')
        opmopt = {'COMPFLAGS="$COMPFLAGS /MD /Qopenmp"'};
        
    elseif contains(cc.ShortName,{'gcc' 'mingw64'}) % not tested
        opmopt = {'CFLAGS="$CFLAGS -fopenmp"'
            'LDFLAGS="$LDFLAGS -fopenmp"'};
    else
        fprintf('Error: Not known compiler\n')
        fprintf('You might want to edit %s if you know how to setup openmp options\n', me)
        return
    end

No idea how to compile (or compatible) with CUDA.

Note that the MSVS OpenMP has some limitation.

[Moein Mozaffarzadeh]

I have just tried to compile outside of Matlab environment to see if the problem is with the compiler or Matlab. I used

g++ OpenMPTest.c -o OpenMPTest.exe -fopenmp

in the windows command bar, and compiled the following code.

#include <stdio.h>
//#include "mex.h"
#include "omp.h"
#include <unistd.h>
//void mexFunction(int nlhs, mxArray* plhs[],
//	int nrhs, const mxArray* prhs[])
int main ()
{
#pragma omp parallel
	{
		printf("Hello(%d), ", omp_get_thread_num());
		printf("World(%d) \n", omp_get_thread_num());
	}
	system("pause");
	return 0;
}

It works well. So, there is nothing wrong with the compiler installation in my Laptop.

I changed the code back to the version the output will be a MEX file as follows:

#include <stdio.h>
#include "mex.h"
#include "omp.h"
#include <unistd.h>
void mexFunction(int nlhs, mxArray* plhs[],
	int nrhs, const mxArray* prhs[])
//int main ()
{
#pragma omp parallel
	{
		printf("Hello(%d), ", omp_get_thread_num());
		printf("World(%d) \n", omp_get_thread_num());
	}
	system("pause");
//	return 0;
}

and now using the following to compile, following your suggestion:

mex -v CFLAGS="$CFLAGS -fopenmp" LDFLAGS="$LDFLAGS -fopenmp" OpenMPTest.c

the code compiles, but my Matlab crashes once i run the .mex file. Please let me know what is it that I'm missing.

Moein.

[Bruno Luong]

I never use "printf" in mex, rather mexPrintf(), but such function are not suitable for parallel calling (notr reentrance).

You must test with something simple.

[Joss Knight]

It certainly could be a problem that you're using printf, which MEX overrides with mexPrintf in order to output to the command window; thus you may be attempting to call into MATLAB from two different threads, which won't be supported.

Try using fprintf(stderr, ...) or just put #undef printf at the top of your file.

I'm guessing attempting a system call in a mexFunction may be inadvisable too, not sure though.

[Joss Knight]

By the way if you were already adding -fopenmp to CFLAGS then that's fine, it won't make any difference adding it to CXXFLAGS. All that really matters is the compiler command you see in the verbose output ( -v ) . As long as the right compiler switches appear in that line (or those lines) you know the right thing is happening. MEX is basically just a MAKE system which helps you invoke your compiler with the right commands and with the correct environment variables set on the system. With a bit of clever interpretation you can reproduce its behaviour using system calls (but getting all the library and include paths right is hard).

[Moein Mozaffarzadeh]

thank you @Joss Knight and @Bruno Luong for your help. you were rigth about printf. So, i changed the code as follows:

#include <stdio.h>
#include "mex.h"
#include "omp.h"
void mexFunction(int nlhs, mxArray* plhs[],
	int nrhs, const mxArray* prhs[])
//int main ()
{
	int* Output;
	plhs[0] = mxCreateNumericMatrix(1, 6, mxINT32_CLASS, mxREAL);
	Output = (int*)mxGetData(plhs[0]);

	omp_set_num_threads(6);
#pragma omp parallel
	{
		int ID = omp_get_thread_num();
		Output[ID] = ID+10;

	}
}

So, as you have noticed. this is a C code. So, I compiled it with the following command in Matlab and it works fine:

mex CFLAGS="$CFLAGS -fopenmp" LDFLAGS="$LDFLAGS -fopenmp" OpenMPTest.c

Now, here is the same code, but in .cu format:

#include <cuda_runtime.h>
#include "device_launch_parameters.h"
#include <stdio.h>
#include "cuda.h"
#include "mex.h"
#include "omp.h"

void mexFunction(int nlhs, mxArray* plhs[],
	int nrhs, const mxArray* prhs[])
{
	int* Output;
	plhs[0] = mxCreateNumericMatrix(1, 6, mxINT32_CLASS, mxREAL);
	Output = (int*)mxGetData(plhs[0]);

	omp_set_num_threads(6);
#pragma omp parallel
	{
		int ID = omp_get_thread_num();
		Output[ID] = ID + 10;

	}
}

Following your comments, I compiled this code with:

mexcuda -v CFLAGS="$CFLAGS -fopenmp" LDFLAGS="$LDFLAGS -fopenmp"  OpenMPTest.cu

When i run this, the output should be "10 11 12 13 14 15" (or something like this in different orders), but I get 10 0 0 0 0 0. So, OpenMP is not working.

Following page 25 of this LINK (says: add "-Xcompiler -fopenmp" flag and -lgomp flag ), I see that I need to use (it mi)

mexcuda -v CFLAGS="$CFLAGS -Xcompiler -fopenmp" LDFLAGS="$LDFLAGS -Xcompiler -fopenmp" -lgomp OpenMPTest.cu

However, I get error for the -lgomp. Matlab cannot find it. I also looked into the Matlab/Mingw base folder, but could not something with such a name.

Could you please let me know what is it that I'm missing here? is the compile command wrong?

Moein.

[Bruno Luong]

Sorry I don't know cuda so I can't help you.

[Walter Roberson]

https://stackoverflow.com/questions/22938238/difference-between-linking-openmp-with-fopenmp-and-lgomp

You should not be linking -lgomp yourself if you use -fopenmp

[Joss Knight]

I can't comment on exactly why nvcc might have trouble building host-side code with OpenMP syntax. You may be better off keeping your host-side code in a .c or .cpp file and putting your device-side code in a separate .cu file. That way, MEXCUDA will invoke gcc for the host code instead of nvcc.

[Bruno Luong]

According to Nvidia site the flag is

nvcc ... -Xcompiler -openmp 

[俊凯 王]

fist of all，i remember in mexfunction you can't printf in a parallel zone

[俊凯 王]

this will make matlab crush

Answer 1

[俊凯 王]

til now i can't find the corroect compile cmd to create a cu project with omp activited

but there are some other ways can handle this trouble,like me i just use omp to create binary tree in cpu,and i want to make this tree root(struct in C) accessible to cu project in matlab

in fact,use intptr_t which cast pointer into int, matlab can accept this int parameter ,next step,we use this int as a input for a matlab cu input parameter

https://www.mathworks.com/matlabcentral/answers/1448829-mex-c-struct-pointer?s_tid=srchtitle

all in all,pointer can be casted into a int parameter for matlab ,and matlab transfer this parameter to cu project , all the variable in CPP/C can sited as a pointer ,this method maybe useful

.cpp(mex64) pointer->int matlab int -> .cu(project) int->pointer