PCT GPU Computing: CUDA error was: setting the device when a process is active is not allowed

2 views (last 30 days)
Hi, everyone!
It seems to me that there is a problem in Parallel Computing Toolbox (PCT) with CUDA operation. The problem is that the setting of CUDA device in PCT fails, if CUDA device has been set up earlier within the same process by someone else, therefore, I cannot use PCT functionality, working with CUDA in my application which is also used by CUDA.
I will give you a simple example. Here it is:
% Starting MATLAB Environment...
% 1. Load a DLL file used by CUDA (or MEX file, it does not matter, only for example)
loadlibrary('MyComputingLib.dll', 'MyComputingLib.h', 'alias', 'cuda_lib')
% 2. perform calculations...
% 3. try to use PCT CUDA...
gpuDevice()
% ... and the error message appears:
An unexpected error occurred during CUDA execution. The CUDA error was: setting the device when a process is active is not allowed.
If the aforementioned actions are performed in the reversed order, the error does not occur (My software works correctly with CUDA device setting). Why cannot MATLAB set up CUDA device, when, starting with CUDA 4.0 version, it is possible to set up CUDA device a multiple number of times within one process?
I think this limitation is the rudiment of what has been left from the old CUDA versions (old derivers), when it was impossible to set up a device a multiple number of times after the process start.
Thank you in advance for your answer.

Accepted Answer

Joss Knight
Joss Knight on 20 Dec 2016
In MATLAB versions up to R2016b, MATLAB's built-in commands must be the first activity on each device within the MATLAB process after a device reset. If you are calling into code that makes a CUDA API call on any device before MATLAB does, MATLAB will error unless you call cudaDeviceReset() on that device before MATLAB tries to access it.
This happens because MATLAB sets device flags when switching to or resetting a device, and this is only allowed before any kernels have run after a reset.
As some have pointed out, if you have only one GPU you can work around this by making sure MATLAB (or any MATLAB workers in a pool) uses the GPU before any of your custom functionality. The more robust solution for your own multi-gpu custom code is to call cudaDeviceReset() on each device you have used before you return control to MATLAB.
This behaviour will no longer occur in the next version of MATLAB.
  3 Comments
Joss Knight
Joss Knight on 21 Dec 2016
That's a surprise. I wouldn't have expected that to happen. Is this the same "Cannot set device on active process" error?
Matt J
Matt J on 21 Dec 2016
Edited: Matt J on 21 Dec 2016
I did not keep track of the error messages, but I don't recall seeing that particular one.
I've been surmising that the GPU isn't distinguishing between MATLAB instances. If external CUDA code in MATLAB session #1 uses the GPU first, then subsequently MATLAB built-ins in session #2 use it, it is as if the same order of events took place in the same session.
Anyway, I would be grateful for some detailed documentation/guidelines on how external CUDA code (using loadlibrary etc...) can interact safely with the Optimization Toolbox and maybe also communicate gpuArray data to each other. Our lab has regular need of this.

Sign in to comment.

More Answers (1)

Matt J
Matt J on 26 Nov 2013
Edited: Matt J on 26 Nov 2013
I seem to be experiencing the same thing. Hope TMW can comment...
However, if what I'm seeing is the same as you, a simple workaround is to do something like this in startup.m
a=gpuArray(1); clear a
I find that as long as gpuArray is the first thing to use the GPU in the MATLAB session, things seem to work normally.
  1 Comment
David Pate
David Pate on 5 Dec 2016
Edited: David Pate on 5 Dec 2016
I also had this problem in Matlab 2016a, and I'm using a solution similar to yours. In my startup file I have:
try
gpu = gpuDevice;
clear gpu;
catch
disp('No GPU device found during startup');
end

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!