pixelLabelImageDatastore not partitionable despite underlying imageDatastore and pixelLabelDatastore are

3 views (last 30 days)
Jari Manni
Jari Manni on 18 Feb 2022
Answered: Joss Knight on 24 Feb 2022
I have a pixelLabelImageDatastore constructed with: pximds = pixelLabelImageDatastore(imds, pxds). Both imds and pxds are partitionable but the resulting pximds is not, which stops me from using multi-gpu.
To me it looks very strange. How do I make a partitionable pixelLabelImageDatastore?
>> isPartitionable(imds)
ans =
>> isPartitionable(pxds)
ans =
>> pximds = pixelLabelImageDatastore(imds, pxds);
>> isPartitionable(pximds)
ans =

Accepted Answer

Joss Knight
Joss Knight on 24 Feb 2022
For parallel training of a semantic segmentation network, you're supposed to use combine to join the two input datastores rather than using a pixelLabelImageDatastore.

More Answers (1)

yanqi liu
yanqi liu on 21 Feb 2022
yes,sir,may be use partitionCamVidData,such as
function [imdsTrain, imdsVal, imdsTest, pxdsTrain, pxdsVal, pxdsTest] = partitionCamVidData(imds,pxds)
% Partition CamVid data by randomly selecting 60% of the data for training. The
% rest is used for testing.
% Set initial random state for example reproducibility.
numFiles = numel(imds.Files);
shuffledIndices = randperm(numFiles);
% Use 60% of the images for training.
numTrain = round(0.60 * numFiles);
trainingIdx = shuffledIndices(1:numTrain);
% Use 20% of the images for validation
numVal = round(0.20 * numFiles);
valIdx = shuffledIndices(numTrain+1:numTrain+numVal);
% Use the rest for testing.
testIdx = shuffledIndices(numTrain+numVal+1:end);
% Create image datastores for training and test.
trainingImages = imds.Files(trainingIdx);
valImages = imds.Files(valIdx);
testImages = imds.Files(testIdx);
imdsTrain = imageDatastore(trainingImages);
imdsVal = imageDatastore(valImages);
imdsTest = imageDatastore(testImages);
% Extract class and label IDs info.
classes = pxds.ClassNames;
labelIDs = camvidPixelLabelIDs();
% Create pixel label datastores for training and test.
trainingLabels = pxds.Files(trainingIdx);
valLabels = pxds.Files(valIdx);
testLabels = pxds.Files(testIdx);
pxdsTrain = pixelLabelDatastore(trainingLabels, classes, labelIDs);
pxdsVal = pixelLabelDatastore(valLabels, classes, labelIDs);
pxdsTest = pixelLabelDatastore(testLabels, classes, labelIDs);
  1 Comment
Jari Manni
Jari Manni on 21 Feb 2022
Hi, thanks for the answer. But I think this is a separate issue. My problem is not on the train set, val set, test set partitioning, but related to multi gpu training where training set is splited into different GPU cards.
When I specify 'ExecutionEnvironment', 'multi-gpu', I get the following error:
'Error using trainNetwork (line 184)
The input datastore is not Partitionable and does not support parallel operations.
UnderlyingCause: [1×1 MException]
identifier: 'nnet_cnn:cnn:GeneralDatastoreDispatcher:NonPartitionableDatastore'
message: 'The input datastore is not Partitionable and does not support parallel operations.'
cause: {}
stack: [2×1 struct]
Correction: []

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!