How do I use an augmentedImageDatastore with DispatchInBackground?
2 views (last 30 days)
Show older comments
I am working on a deep learning CNN problem. I want to use the augmentedImageDatastore to preprocess my images. However, the augmentedImageDatastore is slower than preprocessing the images before hand and storing them on the disk. To speed up the computations, I want to set the DispatchInBackround property to true.
However, when I do this, a parallel pool is created on my machine. Whenever a parallel pool is active, Matlab complains that my GPU device is busy and will refuse to train the network on the GPU (also when the ExecutionEnvironment is set to 'auto', 'gpu', or 'parallel').
I am using Matlab 2018a, with a 960m GPU and i7-6700HQ processor. The number of threads per worker is 1 (I have also tried 2 threads per worker), with 4 workers per pool (I have also tried 2 workers per pool).
The exact error is: Error using trainNetwork (line 154) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable
Error in cnn (line 150) convnet{nested_idx} = trainNetwork(train_Store, layers, options);
Caused by: Error using gpuDevice (line 26) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable
Or, when using parallel (and the pool is started automatically): Starting parallel pool (parpool) using the 'local' profile ... Preserving jobs with IDs: 1 2 3 4 5 6 7 because they contain crash dump files. You can use 'delete(myCluster.Jobs)' to remove all jobs created with profile local. To create 'myCluster' use 'myCluster = parcluster('local')'. connected to 4 workers. Error using trainNetwork (line 154) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable
Error in cnn (line 150) convnet{nested_idx} = trainNetwork(train_Store, layers, options);
Caused by: Error using nnet.internal.cnn.util.GPUShouldBeUsed (line 22) Error detected on worker 2. Error using gpuDevice (line 26) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable
2 Comments
Joss Knight
on 2 Jul 2018
It looks like your driver is configured to only allow one process to access the GPU, which isn't surprising if this is a Windows laptop. Do you have the same issue when you run two instances of MATLAB - is only one of them allowed access to the GPU?
I'm a little surprised that DispatchInBackground can cause this issue. Try starting your pool in advance and disabling the GPU to force the pool to work on the CPU:
parpool(4);
spmd
setenv CUDA_VISIBLE_DEVICES -1;
end
Answers (0)
See Also
Categories
Find more on GPU Computing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!