CUDA_ERROR_UNKNOWN

Hello,

I've seen several people with this question from about a year back, but I still haven't found the solution for my problem. I have written several CUDA kernel functions that run one after another inside a for loop in Matlab with feval and a CUDAKernel object. The code runs fine with smaller amounts of threads, but once I get to a certain total amounts of threads, the code quits and returns:

error during: waiting for kernel invocation to complete: cuda error was: CUDA_ERROR_UNKNOWN

I'm certain that it's not an issue with block size, since it works for smaller amounts of total threads with the same block size. I also tried to split the function that craps out into multiple smaller kernel functions, but it still has the same error. The kernel doesn't run for more than 2 seconds, so it's not a timeout error. Does anybody have a suggestion for what could be wrong?

Thanks, -mrcherry433

[Wajahat]

Try starting matlab with sudo:

>> sudo /usr/local/MATLAB/R2014a/bin/matlab -desktop

[Pramod Rao]

hi wajahat even i am having same error issue in windows .

Answer 1

[Jack]

I've had this error too, on linux (3.11.0-20-generic) after updating the graphics card drivers from nvidia-319 to nvidia-331. I strongly suspect that this error is always related to your cuda drivers.

In my case, starting matlab from the command prompt very neatly told me that my kernel had forgotten to actually modprobe the drivers! A simple

sudo modprobe nvidia-331-uvm

fixed it. This likely came about as I had a rather involved amount of hassle installing and configuring nvidia-drivers. I can now use gpu-accelerated code again. Woo.

One other thing -- the shell from which I start matlab contains the following environment variables set in its .bashrc, which may or may not be required for finding the relevant libraries (arrange to taste if you're not using cuda-6.0):

export CUDA_HOME=/usr/local/cuda-6.0
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}
export PATH=${CUDA_HOME}/bin:${PATH}

Good luck!

Answer 2

[Aaron Meldrum]

I'm seeing this error as well after updating my drivers. I have a Windows 10 system with 4x Tesla K80, and things run fine in 2016b on driver set 356.54, however after updating to Matlab 2018 I've tried a number of driver versions (starting with the earliest to support Cuda 9.0) and continually get the following error when any of my simulations that do heavy computation start

__________________________________

Error using gpuArray/arrayfun An error occurred during PTX compilation of image.

The error log was:

The CUDA error code was: CUDA_ERROR_UNKNOWN

_____________________________________

This error occurs in Matlab 2015, 2016, 2017, and 2018 when using (seemingly) any driver set compatible with them. It does not happen when doing fairly simple GPU array manipulation/multiplication though. I have't tried all of the driver versions, but i've tried several, inlcuding the oldest and newest compatible. I've tried downgrading to 356.54 and running it in 2016b again, and the problem goes away.

Answer 3

[María]

I have a similar one...

An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_UNKNOWN.

Any ideas?

Answer 4

[Joss Knight]

The CUDA runtime is not fabulous about keeping track of the originating error in your kernel and it often falls back to CUDA_ERROR_UNKNOWN. Judging by your description the originating error was probably CUDA_ERROR_LAUNCH_OUT_OF_RESOURCES. Your launch configuration was not compatible with the number of registers/amount of shared memory left on the unused SMs. Try putting in the odd wait(gpuDevice) in your code to see if forcing the kernels to run synchronously fixes the problem.