pixelLabelImageDatastore not partitionable despite underlying imageDatastore and pixelLabelDatastore are

Hi,

I have a pixelLabelImageDatastore constructed with: pximds = pixelLabelImageDatastore(imds, pxds). Both imds and pxds are partitionable but the resulting pximds is not, which stops me from using multi-gpu.

To me it looks very strange. How do I make a partitionable pixelLabelImageDatastore?

>> isPartitionable(imds)

ans =

  logical

   1

>> isPartitionable(pxds)

ans =

  logical

   1

>> pximds = pixelLabelImageDatastore(imds, pxds);
>> isPartitionable(pximds)

ans =

  logical

   0

Answer 1

[Joss Knight]

For parallel training of a semantic segmentation network, you're supposed to use combine to join the two input datastores rather than using a pixelLabelImageDatastore.

Answer 2

[yanqi liu]

yes，sir，may be use partitionCamVidData，such as

https://ww2.mathworks.cn/help/releases/R2019a/vision/examples/semantic-segmentation-using-deep-learning.html

function [imdsTrain, imdsVal, imdsTest, pxdsTrain, pxdsVal, pxdsTest] = partitionCamVidData(imds,pxds)
% Partition CamVid data by randomly selecting 60% of the data for training. The
% rest is used for testing.
    
% Set initial random state for example reproducibility.
rng(0); 
numFiles = numel(imds.Files);
shuffledIndices = randperm(numFiles);

% Use 60% of the images for training.
numTrain = round(0.60 * numFiles);
trainingIdx = shuffledIndices(1:numTrain);

% Use 20% of the images for validation
numVal = round(0.20 * numFiles);
valIdx = shuffledIndices(numTrain+1:numTrain+numVal);

% Use the rest for testing.
testIdx = shuffledIndices(numTrain+numVal+1:end);

% Create image datastores for training and test.
trainingImages = imds.Files(trainingIdx);
valImages = imds.Files(valIdx);
testImages = imds.Files(testIdx);

imdsTrain = imageDatastore(trainingImages);
imdsVal = imageDatastore(valImages);
imdsTest = imageDatastore(testImages);

% Extract class and label IDs info.
classes = pxds.ClassNames;
labelIDs = camvidPixelLabelIDs();

% Create pixel label datastores for training and test.
trainingLabels = pxds.Files(trainingIdx);
valLabels = pxds.Files(valIdx);
testLabels = pxds.Files(testIdx);

pxdsTrain = pixelLabelDatastore(trainingLabels, classes, labelIDs);
pxdsVal = pixelLabelDatastore(valLabels, classes, labelIDs);
pxdsTest = pixelLabelDatastore(testLabels, classes, labelIDs);
end

[Jari Manni]

Hi, thanks for the answer. But I think this is a separate issue. My problem is not on the train set, val set, test set partitioning, but related to multi gpu training where training set is splited into different GPU cards.

When I specify 'ExecutionEnvironment', 'multi-gpu', I get the following error:

'Error using trainNetwork (line 184)

The input datastore is not Partitionable and does not support parallel operations.

UnderlyingCause: [1×1 MException]

identifier: 'nnet_cnn:cnn:GeneralDatastoreDispatcher:NonPartitionableDatastore'

message: 'The input datastore is not Partitionable and does not support parallel operations.'

cause: {}

stack: [2×1 struct]

Correction: []