ROC curve for the validation set

Hi,

I am working on a classification problem, where I use 10-fold cross-validation. I have made the code for the ROC curve for the training part of the data. I want to do the same for the validation set. How can I do that? My code is below:

indices = crossvalind('Kfold',trainingData(:,end),10);

for i = 1:10
    test = (indices == i); 
    train = ~test;
    
    % Linear SVM
    classificationLinearSVM128 = fitcsvm(...
    trainingData(train,1:end-1),...
    trainingData(train,end), ...
    'KernelFunction', 'linear', ...
    'PolynomialOrder', [], ...
    'KernelScale', 'auto', ...
    'BoxConstraint', 1, ...
    'Standardize', true, ...
    'ClassNames', [0; 1]);

    % Training
    [predsLinSVM128train,~] = predict(classificationLinearSVM128,trainingData(train,1:end-1));
    targetsLinSVM128train = trainingData(train,end);
    [~,scoresLinSVM128] = resubPredict(fitPosterior(classificationLinearSVM128));
    [xLinSVM128,yLinSVM128,~,aucLinSVM128] = perfcurve(trainingData(train,end),scoresLinSVM128(:,2),1);
    

    % Validation
    [predsLinSVM128test,~] = predict(classificationLinearSVM128,trainingData(test,1:end-1));
    targetsLinSVM128test = trainingData(test,end);
   

    [~,scoresLinSVM128test] = resubPredict(fitPosterior(classificationLinearSVM128));
    %[xLinSVM128test,yLinSVM128test,~,aucLinSVM128test] = perfcurve(trainingData(test,end),scoresLinSVM128test(:,2),1);

end

figure()
subplot(121)
confusionchart(targetsLinSVM128train,predsLinSVM128train)
title('Linear SVM, training')
subplot(122)
confusionchart(targetsLinSVM128test,predsLinSVM128test)
title('Linear SVM, validation')

figure()
plot(xLinSVM128,yLinSVM128,'LineWidth',2)
xlabel('False Positive Rate')
ylabel('True Positive Rate')
title('ROC, Linear SVM')

I have tried the line that I have commented, but it does not work. Can anyone help with this? The positive class is 1.

Answer 1

[Shashank Gupta]

Hi Uerm,

I don’t see any point, why “perfcurve" function should not work. You even used the same function in your training part and it seems to be working there. I am not really sure why it is not working. Also you can verify it by calculating FPR(False positive rate) and TPR(True positive rate) manually or through ROC function, plot the FPR vs TPR and see if it matches with the perfcurve function output.

Also let me know if you find something interesting. It is supposed to work perfectly. If you are getting any error while using the function post that as well.

[Uerm]

Hi Shashank,

Thanks for your reply. Fortunately, I did manage to make it work for both the training part and the validation part.

[Silvia]

Hi Uerm,

How did you solve the problem?

Thank you