Errors when using gpuArray: "CUSPARSE_​INTERNAL_E​RROR" ,followed by "CUDA-capable devices are busy or unavailable"

I am creating a 400x400 sparse complex double matrix, and then trying to convert it to a gpuArray. However, the conversion results in the following error upon the first attempt after a restart of Matlab:

Error using gpuArray
An unexpected error occurred on the device. The error code was: CUSPARSE_INTERNAL_ERROR.

Once this error was thrown, if I try again to convert the matrix a second time (without a restart of Matlab), then error is now:

Error using gpuDevice (line 26)
An unexpected error occurred during CUDA execution. The CUDA error was:
all CUDA-capable devices are busy or unavailable

If I restart Matlab, the first error returns, and the cycle repeats. Does anyone know how this can be fixed?

_____________________________________________________________________________________________

The graphics card is the Nvidia Quadro P4000, with the latest driver (27.21.14.6109 from here), and I am running Matlab R2017a on Windows 10 64-bit. The ouput from running gpuDevice directly after a Matlab restart is as follows:

 CUDADevice with properties:

                      Name: 'Quadro P4000'
                     Index: 1
         ComputeCapability: '6.1'
            SupportsDouble: 1
             DriverVersion: 11.2000
            ToolkitVersion: 8
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 8.5899e+09
           AvailableMemory: 7.5037e+09
       MultiprocessorCount: 14
              ClockRateKHz: 1480000
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

I have also tried increasing the TdrDelay registry key to 8 seconds in the Windows registry, as described here, and the problem persists.

[Joss Knight]

This does look like a bug, but it's not specific enough to investigate. Can you provide the code that generates the problematic matrix? Or upload a MAT-file with it in?

If you have access to a later version of MATLAB, do check to see whether the problem is fixed.

You may be able to recover use of your device without restarting MATLAB by resetting the device via a call to gpuDevice(1).

[Thomas Barrett]

Hi @Joss Knight

I did as you suggested and upgraded to R2020b. The problem has disappeared, and everything works correctly.

Thanks for the support.

Answer 1

[Thomas Barrett]

This was fixed when upgrading to a more recent version of Matlab (R2020b).