CUDA_ERROR​_CONTEXT_I​S_DESTROYE​D when calling gpuArray after simulations on GPU

I run some simulations with MCXLab, a toolbox using the GPU, that perform correctly. Then I want to do some matrix operations on the extracted data, and I do that iteratively. Here is a simplified code:

for ii=1:N

MCXLab simulations => OK

D=some matrix of size (N);

D=gpuArray(D);

update simulation parameters

end

The weird thing is that it performs ok for ii=1 but throws this error at D=gpuArray(D);:

"Error using gpuArray

Encountered unexpected error during CUDA execution. The CUDA error was:

CUDA_ERROR_CONTEXT_IS_DESTROYED". for the second iteration.

Once the error is thrown, launching again the entire code gives the same error at ii=1 (but MCXlab simulations are done correctly).

Trying reset(gpuDevice()) makes matlab crash ('Matlab stopped working and needs to close') with the warning

"Warning: Encountered unexpected error during CUDA execution. The CUDA error was:

CUDA_ERROR_CONTEXT_IS_DESTROYED "

The code is running on Windows, Matlab R2021a, with a Nvidia Quadro P2000 GPU. MCXLab is a Mex open source Light simulator.

The error happens whetever the size N (even a scalar), with single or double precision.

[William]

I seem to be having the same issue, after using mcxlab any further computations on the GPU throw back this error!

[Charlotte Constans]

I just solved this problem by updating mcxlab with the latest compiled code (https://github.com/fangq/mcx). You just need reset(gpuDevice) between mcxlab simulations and further GPU calculations.

Answer 1

[Joss Knight]

Probably MCXLab is doing something bad to your GPU. Try adding wait(gpuDevice) after your MCXLab code.

[Charlotte Constans]

With wait(gpuDevice) , I get this error: "Error using gpuArray

Encountered unexpected error during CUDA execution. The CUDA error was:

CUDA_ERROR_INVALID_VALUE"

on the second call of gpuArray.

for ii=1:N

MCXLab simulations => OK

wait(gpuDevice);

D=some matrix of size (N); X=some other matrix;

D=gpuArray(D);

X=gpuArray(X);

update simulation parameters

end

the error appears only for X, so D=gpuArray(D); is done correctly.

If I try without X, only D, the code runs but with this warning: "Warning: Encountered unexpected error during CUDA execution. The CUDA error was:

CUDA_ERROR_INVALID_VALUE " starting from ii=2 (no warning on the first iteration), for the line D=gpuArray(D);

[Joss Knight]

Very confusing. Can you do the same with the profiler on? i.e.

profile on
myScript

Basically, it's clear that there is a fault in MCXLab so it's going to be pretty hard to debug that for you. You may have to take this to the MCXLab forum.

An external tool making use of the GPU could easily be doing things incompatible with MATLAB even if it doesn't have an explicit bug. For instance, it could be failing to synchronize, resetting the device, changing runtime settings etc.

[Charlotte Constans]

Here are the profile results (screenshot of profile viewer and zipped htlm files).

[Joss Knight]

Hiya, I don't want the profile results, I just want to see what the behaviour is like when the profile is on.

[Charlotte Constans]

Oh ok, the behaviour does not change with the profile on

[Joss Knight]

Are you selecting the device before you call MCXLab? Try calling gpuDevice() at the start of your script.

[Charlotte Constans]

Yes, nothing changes

[Joss Knight]

After restarting MATLAB?

[Charlotte Constans]

After restarting matlab, the error happens for the second iteration ii=2, for X (and not D).

After that, without restarting, the error happens at first iteration for X.

I just realized that the order of X and D does not matter, but apparently the size is the important criteria. The error happens BELOW a critical size that I evaluated to be 64800 bytes (X=zeros(90,'double') and below throws the error, X=zeros(91,'double') and above is ok).

[Joss Knight]

So you kept the code to call gpuDevice() before you do anything else, restarted MATLAB, and the issue still happens?

I'm afraid you'll have to talk to the MCXLab forums. This is an issue caused by MCXLab, either resetting the device or doing something illegal internally. You could tell them that calling MCXLab with given inputs put the device into a bad state and MATLAB can no longer use it.

[Charlotte Constans]

Yes it still happens.

Thank you very much for your help, I will contact the MCXLab team.