Compute additional classification performance metrics

Since R2022a

## Syntax

``UpdatedROCObj = addMetrics(rocObj,metrics)``

## Description

`rocmetrics` computes the false positive rates (FPR), true positive rates (TPR), and additional metrics specified by the `AdditionalMetrics` name-value argument. After creating a `rocmetrics` object, you can compute additional classification performance metrics by using the `addMetrics` function.

example

````UpdatedROCObj = addMetrics(rocObj,metrics)` computes additional classification performance metrics specified in `metrics` using the classification model information stored in the `rocmetrics` object `rocObj`.`UpdatedROCObj` contains all the information in `rocObj` plus additional performance metrics computed by `addMetrics`. The function attaches the additional computed metrics (`metrics`) as new variables in the table of the `Metrics` property.If you compute confidence intervals when you create `rocObj`, the `addMetrics` function computes the confidence intervals for the additional `metrics`. The new variables in the `Metrics` property contain a three-column matrix in which the first column corresponds to the metric values, and the second and third columns correspond to the lower and upper bounds, respectively. ```

## Examples

collapse all

Compute the performance metrics (FPR, TPR, and expected cost) for a multiclass classification problem when you create a `rocmetrics` object. Compute additional metrics, the positive predictive value (PPV) and the negative predictive value (NPV), and add them to the object.

Load the `fisheriris` data set. The matrix `meas` contains flower measurements for 150 different flowers. The vector `species` lists the species for each flower. `species` contains three distinct flower names.

`load fisheriris`

Train a classification tree that classifies observations into one of the three labels. Cross-validate the model using 10-fold cross-validation.

```rng("default") % For reproducibility Mdl = fitctree(meas,species,Crossval="on");```

Compute the classification scores for validation-fold observations.

```[~,Scores] = kfoldPredict(Mdl); size(Scores)```
```ans = 1×2 150 3 ```

`Scores` is a matrix of size `150`-by-`3`. The column order of `Scores` follows the class order in `Mdl`. Display the class order stored in `Mdl.ClassNames`.

`Mdl.ClassNames`
```ans = 3x1 cell {'setosa' } {'versicolor'} {'virginica' } ```

Create a `rocmetrics` object by using the true labels in `species` and the classification scores in `Scores`. Specify the column order of `Scores` using `Mdl.ClassNames`. By default, `rocmetrics` computes the FPR and TPR. Specify `AdditionalMetrics="ExpectedCost"` to compute the expected cost as well.

```rocObj = rocmetrics(species,Scores,Mdl.ClassNames, ... AdditionalMetrics="ExpectedCost");```

The table in the `Metrics` property of `rocObj` contains performance metric values for all three classes, vertically concatenated according to the class order. Find and display the rows for the second class in the table.

```idx = strcmp(rocObj.Metrics.ClassName,Mdl.ClassNames(2)); rocObj.Metrics(idx,:)```
```ans=13×5 table ClassName Threshold FalsePositiveRate TruePositiveRate ExpectedCost ______________ _________ _________________ ________________ ____________ {'versicolor'} 1 0 0 0.074074 {'versicolor'} 1 0.01 0.7 0.023704 {'versicolor'} 0.95455 0.02 0.8 0.017778 {'versicolor'} 0.91304 0.03 0.9 0.011852 {'versicolor'} -0.2 0.04 0.9 0.013333 {'versicolor'} -0.33333 0.06 0.9 0.016296 {'versicolor'} -0.6 0.08 0.9 0.019259 {'versicolor'} -0.86957 0.12 0.92 0.023704 {'versicolor'} -0.91111 0.16 0.96 0.026667 {'versicolor'} -0.95122 0.31 0.96 0.048889 {'versicolor'} -0.95238 0.38 0.98 0.057778 {'versicolor'} -0.95349 0.44 0.98 0.066667 {'versicolor'} -1 1 1 0.14815 ```

The table in `Metrics` contains the variables for the class names, threshold, false positive rate, true positive rate, and expected cost (the additional metric).

After creating a `rocmetrics` object, you can compute additional metrics using the classification model information stored in the object. Compute the PPV and NPV by using the `addMetrics` function. To overwrite the input argument `rocObj`, assign the output of `addMetrics` to the input.

`rocObj = addMetrics(rocObj,["PositivePredictiveValue","NegativePredictiveValue"]);`

Display the `Metrics` property.

`rocObj.Metrics(idx,:)`
```ans=13×7 table ClassName Threshold FalsePositiveRate TruePositiveRate ExpectedCost PositivePredictiveValue NegativePredictiveValue ______________ _________ _________________ ________________ ____________ _______________________ _______________________ {'versicolor'} 1 0 0 0.074074 NaN 0.66667 {'versicolor'} 1 0.01 0.7 0.023704 0.97222 0.86842 {'versicolor'} 0.95455 0.02 0.8 0.017778 0.95238 0.90741 {'versicolor'} 0.91304 0.03 0.9 0.011852 0.9375 0.95098 {'versicolor'} -0.2 0.04 0.9 0.013333 0.91837 0.9505 {'versicolor'} -0.33333 0.06 0.9 0.016296 0.88235 0.94949 {'versicolor'} -0.6 0.08 0.9 0.019259 0.84906 0.94845 {'versicolor'} -0.86957 0.12 0.92 0.023704 0.7931 0.95652 {'versicolor'} -0.91111 0.16 0.96 0.026667 0.75 0.97674 {'versicolor'} -0.95122 0.31 0.96 0.048889 0.60759 0.97183 {'versicolor'} -0.95238 0.38 0.98 0.057778 0.56322 0.98413 {'versicolor'} -0.95349 0.44 0.98 0.066667 0.52688 0.98246 {'versicolor'} -1 1 1 0.14815 0.33333 NaN ```

The table in `Metrics` now includes the `PositivePredictiveValue` and `NegativePredictiveValue` variables in the last two columns, in the order you specified. Note that the positive predictive value (`PPV = TP/(TP+FP)`) is `NaN` for the reject-all threshold (largest threshold), and the negative predictive value (`NPV = TN/(TN+FN)`) is `NaN` for the accept-all threshold (lowest threshold). `TP`, `FP`, `TN`, and `FN` represent the number of true positives, false positives, true negatives, and false negatives, respectively.

## Input Arguments

collapse all

Object evaluating classification performance, specified as a `rocmetrics` object.

Additional model performance metrics to compute, specified as a character vector or string scalar of the built-in metric name, string array of names, function handle (`@metricName`), or cell array of names or function handles. A `rocmetrics` object always computes the false positive rates (FPR) and the true positive rates (TPR) to obtain a ROC curve. Therefore, you do not have to specify to compute FPR and TPR.

• Built-in metrics — Specify one of the following built-in metric names by using a character vector or string scalar. You can specify more than one by using a string array.

NameDescription
`"TruePositives"` or `"tp"`Number of true positives (TP)
`"FalseNegatives"` or `"fn"`Number of false negatives (FN)
`"FalsePositives"` or `"fp"`Number of false positives (FP)
`"TrueNegatives"` or `"tn"`Number of true negatives (TN)
`"SumOfTrueAndFalsePositives"` or `"tp+fp"`Sum of TP and FP
`"RateOfPositivePredictions"` or `"rpp"`Rate of positive predictions (RPP), `(TP+FP)/(TP+FN+FP+TN)`
`"RateOfNegativePredictions"` or `"rnp"`Rate of negative predictions (RNP), `(TN+FN)/(TP+FN+FP+TN)`
`"Accuracy"` or `"accu"`Accuracy, `(TP+TN)/(TP+FN+FP+TN)`
`"FalseNegativeRate"`, `"fnr"`, or `"miss"`False negative rate (FNR), or miss rate, `FN/(TP+FN)`
`"TrueNegativeRate"`, `"tnr"`, or `"spec"`True negative rate (TNR), or specificity, `TN/(TN+FP)`
`"PositivePredictiveValue"`, `"ppv"`, or `"prec"`Positive predictive value (PPV), or precision, `TP/(TP+FP)`
`"NegativePredictiveValue"` or `"npv"`Negative predictive value (NPV), `TN/(TN+FN)`
`"ExpectedCost"` or `"ecost"`

Expected cost, `(TP*cost(P|P)+FN*cost(N|P)+FP*cost(P|N)+TN*cost(N|N))/(TP+FN+FP+TN)`, where `cost` is a 2-by-2 misclassification cost matrix containing `[0,cost(N|P);cost(P|N),0]`. `cost(N|P)` is the cost of misclassifying a positive class (`P`) as a negative class (`N`), and `cost(P|N)` is the cost of misclassifying a negative class as a positive class.

The software converts the `K`-by-`K` matrix specified by the `Cost` name-value argument of `rocmetrics` to a 2-by-2 matrix for each one-versus-all binary problem. For details, see Misclassification Cost Matrix.

The software computes the scale vector using the prior class probabilities (`Prior`) and the number of classes in `Labels`, and then scales the performance metrics according to this scale vector. For details, see Performance Metrics.

• Custom metric — Specify a custom metric by using a function handle. A custom function that returns a performance metric must have this form:

`metric = customMetric(C,scale,cost)`

• The output argument `metric` is a scalar value.

• A custom metric is a function of the confusion matrix (`C`), scale vector (`scale`), and cost matrix (`cost`). The software finds these input values for each one-versus-all binary problem. For details, see Performance Metrics.

• `C` is a `2`-by-`2` confusion matrix consisting of `[TP,FN;FP,TN]`.

• `scale` is a `2`-by-`1` scale vector.

• `cost` is a `2`-by-`2` misclassification cost matrix.

The software does not support cross-validation for a custom metric. Instead, you can specify to use bootstrap when you create a `rocmetrics` object.

Note that the positive predictive value (PPV) is `NaN` for the reject-all threshold for which `TP` = `FP` = `0`, and the negative predictive value (NPV) is `NaN` for the accept-all threshold for which `TN` = `FN` = `0`. For more details, see Thresholds, Fixed Metric, and Fixed Metric Values.

Example: `["Accuracy","PositivePredictiveValue"]`

Example: `{"Accuracy",@m1,@m2}` specifies the accuracy metric and the custom metrics `m1` and `m2` as additional metrics. `addMetrics` stores the custom metric values as variables named `CustomMetric1` and `CustomMetric2` in the `Metrics` property.

Data Types: `char` | `string` | `cell` | `function_handle`

## Output Arguments

collapse all

Object evaluating classification performance, returned as a `rocmetrics` object.

To overwrite the input argument `rocObj`, assign the output of `addMetrics` to `rocObj`:

`rocObj = addMetrics(rocObj,metrics);`

## Version History

Introduced in R2022a