Confusion Matrix of cross validation of an ECOC SVM classifier

Hi, I am using MATLAB 2015 and statistics and machine learning toolbox. I want to do a 10-fold cross validation for an ECOC svm classifier with 19 classes. I want to report test result by obtaining confusion matrix. I used following piece of code:

 template = templateSVM('KernelFunction', 'rbf',  'KernelScale', 'auto', 'BoxConstraint', 1, 'Standardize', 1);
 trainedClassifier = fitcecoc(Features, Labels, 'Learners', template, 'Coding', 'onevsone');
 partModel = crossval(trainedClassifier, 'KFold', 10);
 Accuracy = 1 - kfoldLoss(partModel, 'LossFun', 'ClassifError');
 [validationPredictions, validationScores] = kfoldPredict(partModel);
 confmat=confusionmat(Features(:,1),validationPredictions);

Is the confmat, the average confusion matrix of 10 folds which are taken out during cross validation? I mean, should I use this confusion matrix as test performance of classifier? If it is not, what should I do for doing a test on the performance of classifier? Thanks in advance Mehrdad

[Mehrdad Heydarzadeh]

Thanks Tom for replying, Yes my target labels are in first column of Features. My question is regarding 10-fold cross validation. In 10-fold cv I have 10 classifiers and each of them is tested by 1/10 data. then the accuracy are gathered for calculating confmat. I can ask my question like this: Does the command "crossval" in third line do the cross validation with all 10 classifiers? If this is the case the variable partModel should contain all designed classifiers inside and all data. Right? Thanks in advance, Mehrdad

[Angga Lisdiyanto]

I need the answer too.

Please comment into Tom 's Answer, not to your Question.

[Pallavi]

same query

Answer 1

[Daniel]

Yes. If you examine the crossval object you will see an attribute called Trained. It contains k classifiers where k is the number of folds you specified. Each classifier trains on its respective partition which is specified by the Partition attribute of the crossval object. The kFoldPredict function returns the aggregate predictions of all k folds into the validationPredictions variable so your confusion matrix shows the classification results of all of your data.

For nice plots, use plotconfusion function. If your labels are non-numeric I have had good luck with the heatmap plotting toolbox which can be found in FileExchange

Answer 2

[Tom Lane]

The confusion matrix is one measure of classifier accuracy. You should supply the confmat function with the known labels and the predicted values. It appears you want to supply Labels in place of Features(:,1). Your other argument looks okay; the predicted value for row J is computed using the classifier for which row J is part of the 10% held out, and the training is done on the other 90%.

[Angga Lisdiyanto]

Hi Tom, how to add x & y labels on confusionmat function? And my class value type is string, i cannot use confusionmat function.