Automatically select classification model with optimized hyperparameters
Given predictor and response data, fitcauto
automatically
tries a selection of classification model types with different hyperparameter values. The
function uses Bayesian optimization to select models and their hyperparameter values, and
computes the cross-validation classification error for each model. After the optimization is
complete, fitcauto
returns the model, trained on the entire data set,
that is expected to best classify new data. You can use the predict
and
loss
object functions of the returned model to classify new data and
compute the test set classification error, respectively.
Use fitcauto
when you are uncertain which classifier types best suit
your data. For information on alternative methods for tuning hyperparameters of classification
models, see Alternative Functionality.
returns a classification model Mdl
= fitcauto(Tbl
,ResponseVarName
)Mdl
with tuned hyperparameters. The
table Tbl
contains the predictor variables and the response variable,
where ResponseVarName
is the name of the response variable.
specifies options using one or more name-value pair arguments in addition to any of the
input argument combinations in previous syntaxes. For example, use the
Mdl
= fitcauto(___,Name,Value
)HyperparameterOptimizationOptions
name-value pair argument to
specify how the Bayesian optimization is performed.
[
additionally returns Mdl
,OptimizationResults
] = fitcauto(___)OptimizationResults
, a
BayesianOptimization
object containing the results of the model
selection and hyperparameter tuning process.
Use fitcauto
to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in a table.
Load Data
Load the carbig
data set, which contains measurements of cars made in the 1970s and early 1980s.
load carbig
Categorize the cars based on whether they were made in the USA.
Origin = categorical(cellstr(Origin)); Origin = mergecats(Origin,{'France','Japan','Germany', ... 'Sweden','Italy','England'},'NotUSA');
Create a table containing the predictor variables Acceleration
, Displacement
, and so on, as well as the response variable Origin
.
cars = table(Acceleration,Displacement,Horsepower, ...
Model_Year,MPG,Weight,Origin);
Partition Data
Partition the data into training and test sets. Use approximately 80% of the observations for the model selection and hyperparameter tuning process, and 20% of the observations to test the performance of the final model returned by fitcauto
. Use cvpartition
to partition the data.
rng('default') % For reproducibility of the data partition c = cvpartition(Origin,'Holdout',0.2); trainingIdx = training(c); % Training set indices carsTrain = cars(trainingIdx,:); testIdx = test(c); % Test set indices carsTest = cars(testIdx,:);
Run fitcauto
Pass the training data to fitcauto
. By default, fitcauto
determines appropriate model types to try, uses Bayesian optimization to find good hyperparameter values, and returns a trained model Mdl
with the best expected performance. Additionally, fitcauto
provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
Expect this process to take some time. To speed up the optimization process, consider specifying to run the optimization in parallel, if you have a Parallel Computing Toolbox™ license. To do so, pass 'HyperparameterOptimizationOptions',struct('UseParallel',true)
to fitcauto
as a name-value pair argument.
Mdl = fitcauto(carsTrain,'Origin');
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Learner types to explore: ensemble, knn, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 150 Total time (MaxTime): Inf |=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 1 | Best | 0.14154 | 10.563 | 0.14154 | 0.14154 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 201 | | | | | | | | | MinLeafSize: 7 |
| 2 | Accept | 0.18269 | 0.57392 | 0.14154 | 0.14154 | knn | NumNeighbors: 3 |
| 3 | Accept | 0.23397 | 0.1264 | 0.14154 | 0.14154 | knn | NumNeighbors: 91 |
| 4 | Accept | 0.16308 | 12.867 | 0.14154 | 0.15468 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 274 | | | | | | | | | MinLeafSize: 15 |
| 5 | Accept | 0.20833 | 0.124 | 0.14154 | 0.15468 | knn | NumNeighbors: 4 |
| 6 | Accept | 0.22115 | 0.079641 | 0.14154 | 0.15468 | knn | NumNeighbors: 28 |
| 7 | Accept | 0.16923 | 0.20013 | 0.14154 | 0.15468 | tree | MinLeafSize: 105 |
| 8 | Accept | 0.37179 | 0.59222 | 0.14154 | 0.15468 | svm | BoxConstraint: 0.022186 | | | | | | | | | KernelScale: 0.085527 |
| 9 | Accept | 0.37179 | 0.11659 | 0.14154 | 0.15468 | svm | BoxConstraint: 0.045899 | | | | | | | | | KernelScale: 0.0024758 |
| 10 | Accept | 0.24615 | 0.98386 | 0.14154 | 0.15468 | nb | DistributionNames: kernel | | | | | | | | | Width: 1.1327 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 11 | Accept | 0.16923 | 0.079996 | 0.14154 | 0.15468 | tree | MinLeafSize: 78 |
| 12 | Accept | 0.26923 | 0.10923 | 0.14154 | 0.15468 | svm | BoxConstraint: 11.063 | | | | | | | | | KernelScale: 15.114 |
| 13 | Best | 0.12923 | 0.11568 | 0.12923 | 0.15468 | tree | MinLeafSize: 3 |
| 14 | Accept | 0.21154 | 0.084406 | 0.12923 | 0.15468 | knn | NumNeighbors: 2 |
| 15 | Accept | 0.14154 | 0.080022 | 0.12923 | 0.15294 | tree | MinLeafSize: 1 |
| 16 | Accept | 0.14769 | 0.092395 | 0.12923 | 0.15097 | tree | MinLeafSize: 2 |
| 17 | Accept | 0.14154 | 10.869 | 0.12923 | 0.14872 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 208 | | | | | | | | | MinLeafSize: 10 |
| 18 | Accept | 0.37179 | 0.12386 | 0.12923 | 0.14872 | svm | BoxConstraint: 116.46 | | | | | | | | | KernelScale: 0.52908 |
| 19 | Accept | 0.22769 | 0.15545 | 0.12923 | 0.14872 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 20 | Accept | 0.22115 | 0.070813 | 0.12923 | 0.14872 | knn | NumNeighbors: 8 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 21 | Accept | 0.37179 | 0.11553 | 0.12923 | 0.14872 | svm | BoxConstraint: 45.341 | | | | | | | | | KernelScale: 0.76949 |
| 22 | Accept | 0.12923 | 0.080362 | 0.12923 | 0.14194 | tree | MinLeafSize: 3 |
| 23 | Best | 0.10154 | 0.079656 | 0.10154 | 0.13213 | tree | MinLeafSize: 5 |
| 24 | Accept | 0.22769 | 0.2529 | 0.10154 | 0.13213 | nb | DistributionNames: kernel | | | | | | | | | Width: 0.42571 |
| 25 | Accept | 0.11385 | 0.080085 | 0.10154 | 0.1289 | tree | MinLeafSize: 11 |
| 26 | Accept | 0.13782 | 0.092228 | 0.10154 | 0.1289 | svm | BoxConstraint: 9.7286 | | | | | | | | | KernelScale: 293.41 |
| 27 | Accept | 0.22769 | 0.073346 | 0.10154 | 0.1289 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 28 | Accept | 0.21795 | 0.074914 | 0.10154 | 0.1289 | knn | NumNeighbors: 42 |
| 29 | Accept | 0.24308 | 0.27621 | 0.10154 | 0.1289 | nb | DistributionNames: kernel | | | | | | | | | Width: 4.4662 |
| 30 | Accept | 0.16308 | 12.328 | 0.10154 | 0.1289 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 267 | | | | | | | | | MinLeafSize: 131 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 31 | Accept | 0.24308 | 0.22334 | 0.10154 | 0.1289 | nb | DistributionNames: kernel | | | | | | | | | Width: 0.66296 |
| 32 | Accept | 0.22115 | 0.066711 | 0.10154 | 0.1289 | knn | NumNeighbors: 28 |
| 33 | Accept | 0.13846 | 0.079934 | 0.10154 | 0.12465 | tree | MinLeafSize: 25 |
| 34 | Accept | 0.21474 | 0.085438 | 0.10154 | 0.12465 | knn | NumNeighbors: 14 |
| 35 | Accept | 0.16615 | 10.05 | 0.10154 | 0.12465 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 215 | | | | | | | | | MinLeafSize: 13 |
| 36 | Accept | 0.14154 | 12.866 | 0.10154 | 0.12465 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 254 | | | | | | | | | MinLeafSize: 31 |
| 37 | Accept | 0.22769 | 0.070251 | 0.10154 | 0.12465 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 38 | Accept | 0.37179 | 0.077924 | 0.10154 | 0.12465 | svm | BoxConstraint: 0.0073633 | | | | | | | | | KernelScale: 774.33 |
| 39 | Accept | 0.16923 | 0.068411 | 0.10154 | 0.12552 | tree | MinLeafSize: 82 |
| 40 | Accept | 0.20833 | 0.064563 | 0.10154 | 0.12552 | knn | NumNeighbors: 4 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 41 | Accept | 0.16308 | 12.932 | 0.10154 | 0.12552 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 274 | | | | | | | | | MinLeafSize: 150 |
| 42 | Accept | 0.22462 | 0.24365 | 0.10154 | 0.12552 | nb | DistributionNames: kernel | | | | | | | | | Width: 121.64 |
| 43 | Accept | 0.20308 | 11.027 | 0.10154 | 0.12552 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 229 | | | | | | | | | MinLeafSize: 117 |
| 44 | Accept | 0.16923 | 0.069279 | 0.10154 | 0.12291 | tree | MinLeafSize: 84 |
| 45 | Accept | 0.22769 | 0.078716 | 0.10154 | 0.12291 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 46 | Accept | 0.22769 | 0.068458 | 0.10154 | 0.12291 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 47 | Accept | 0.16615 | 9.9849 | 0.10154 | 0.12291 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 212 | | | | | | | | | MinLeafSize: 49 |
| 48 | Accept | 0.14769 | 14.541 | 0.10154 | 0.12291 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 288 | | | | | | | | | MinLeafSize: 25 |
| 49 | Accept | 0.23077 | 0.21379 | 0.10154 | 0.12291 | nb | DistributionNames: kernel | | | | | | | | | Width: 73.249 |
| 50 | Accept | 0.37179 | 0.091937 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.0036501 | | | | | | | | | KernelScale: 1.0504 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 51 | Accept | 0.21474 | 0.098808 | 0.10154 | 0.12291 | svm | BoxConstraint: 64.859 | | | | | | | | | KernelScale: 23.779 |
| 52 | Accept | 0.37179 | 0.10415 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.16622 | | | | | | | | | KernelScale: 4.4901 |
| 53 | Accept | 0.25846 | 0.2444 | 0.10154 | 0.12291 | nb | DistributionNames: kernel | | | | | | | | | Width: 0.079498 |
| 54 | Accept | 0.21154 | 0.074835 | 0.10154 | 0.12291 | knn | NumNeighbors: 2 |
| 55 | Accept | 0.13846 | 12.173 | 0.10154 | 0.12291 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | MinLeafSize: 8 |
| 56 | Accept | 0.36538 | 0.10958 | 0.10154 | 0.12291 | svm | BoxConstraint: 271.6 | | | | | | | | | KernelScale: 2.743 |
| 57 | Accept | 0.16615 | 11.482 | 0.10154 | 0.12291 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 248 | | | | | | | | | MinLeafSize: 117 |
| 58 | Accept | 0.37179 | 0.095419 | 0.10154 | 0.12291 | svm | BoxConstraint: 7.5785 | | | | | | | | | KernelScale: 0.0066815 |
| 59 | Accept | 0.37179 | 0.097469 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.0017765 | | | | | | | | | KernelScale: 0.86786 |
| 60 | Accept | 0.37179 | 0.11284 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.011465 | | | | | | | | | KernelScale: 0.02747 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 61 | Accept | 0.11692 | 0.077515 | 0.10154 | 0.12239 | tree | MinLeafSize: 12 |
| 62 | Accept | 0.29167 | 0.091617 | 0.10154 | 0.12239 | svm | BoxConstraint: 11.939 | | | | | | | | | KernelScale: 11.002 |
| 63 | Accept | 0.21795 | 0.067171 | 0.10154 | 0.12239 | knn | NumNeighbors: 6 |
| 64 | Accept | 0.18269 | 0.062887 | 0.10154 | 0.12239 | knn | NumNeighbors: 3 |
| 65 | Accept | 0.12923 | 0.075704 | 0.10154 | 0.11989 | tree | MinLeafSize: 3 |
| 66 | Accept | 0.16923 | 0.065889 | 0.10154 | 0.12048 | tree | MinLeafSize: 56 |
| 67 | Accept | 0.1891 | 0.068215 | 0.10154 | 0.12048 | knn | NumNeighbors: 1 |
| 68 | Accept | 0.13231 | 14.135 | 0.10154 | 0.12048 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 270 | | | | | | | | | MinLeafSize: 4 |
| 69 | Accept | 0.22769 | 0.060902 | 0.10154 | 0.12048 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 70 | Accept | 0.37231 | 0.24511 | 0.10154 | 0.12048 | nb | DistributionNames: kernel | | | | | | | | | Width: 1629.5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 71 | Accept | 0.16923 | 0.069135 | 0.10154 | 0.11947 | tree | MinLeafSize: 61 |
| 72 | Accept | 0.22769 | 0.060552 | 0.10154 | 0.11947 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 73 | Accept | 0.16308 | 10.133 | 0.10154 | 0.11947 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 217 | | | | | | | | | MinLeafSize: 70 |
| 74 | Accept | 0.13231 | 13.055 | 0.10154 | 0.11947 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 257 | | | | | | | | | MinLeafSize: 2 |
| 75 | Accept | 0.21474 | 0.069312 | 0.10154 | 0.11947 | knn | NumNeighbors: 49 |
| 76 | Accept | 0.13846 | 0.083714 | 0.10154 | 0.1214 | tree | MinLeafSize: 25 |
| 77 | Accept | 0.12 | 0.069041 | 0.10154 | 0.11923 | tree | MinLeafSize: 6 |
| 78 | Accept | 0.10154 | 0.077399 | 0.10154 | 0.1118 | tree | MinLeafSize: 5 |
| 79 | Accept | 0.12 | 0.076269 | 0.10154 | 0.11007 | tree | MinLeafSize: 4 |
| 80 | Accept | 0.10154 | 0.09325 | 0.10154 | 0.10878 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 81 | Accept | 0.10154 | 0.074541 | 0.10154 | 0.10737 | tree | MinLeafSize: 5 |
| 82 | Accept | 0.12 | 0.069929 | 0.10154 | 0.1063 | tree | MinLeafSize: 4 |
| 83 | Accept | 0.10154 | 0.072591 | 0.10154 | 0.10514 | tree | MinLeafSize: 5 |
| 84 | Accept | 0.10154 | 0.071254 | 0.10154 | 0.10366 | tree | MinLeafSize: 5 |
| 85 | Accept | 0.10154 | 0.077378 | 0.10154 | 0.10361 | tree | MinLeafSize: 5 |
| 86 | Accept | 0.10154 | 0.070643 | 0.10154 | 0.10348 | tree | MinLeafSize: 5 |
| 87 | Accept | 0.12 | 0.070551 | 0.10154 | 0.10286 | tree | MinLeafSize: 4 |
| 88 | Accept | 0.12 | 0.078438 | 0.10154 | 0.1029 | tree | MinLeafSize: 6 |
| 89 | Accept | 0.10154 | 0.072996 | 0.10154 | 0.10262 | tree | MinLeafSize: 5 |
| 90 | Accept | 0.10154 | 0.078162 | 0.10154 | 0.10246 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 91 | Accept | 0.10154 | 0.07038 | 0.10154 | 0.10267 | tree | MinLeafSize: 5 |
| 92 | Accept | 0.10154 | 0.07752 | 0.10154 | 0.10257 | tree | MinLeafSize: 5 |
| 93 | Accept | 0.10154 | 0.076155 | 0.10154 | 0.10217 | tree | MinLeafSize: 5 |
| 94 | Accept | 0.10154 | 0.075983 | 0.10154 | 0.10221 | tree | MinLeafSize: 5 |
| 95 | Accept | 0.10154 | 0.07407 | 0.10154 | 0.10211 | tree | MinLeafSize: 5 |
| 96 | Accept | 0.10154 | 0.080633 | 0.10154 | 0.10207 | tree | MinLeafSize: 5 |
| 97 | Accept | 0.10154 | 0.086164 | 0.10154 | 0.10205 | tree | MinLeafSize: 5 |
| 98 | Accept | 0.10154 | 0.080264 | 0.10154 | 0.10191 | tree | MinLeafSize: 5 |
| 99 | Accept | 0.12308 | 0.076015 | 0.10154 | 0.1021 | tree | MinLeafSize: 17 |
| 100 | Accept | 0.10154 | 0.074579 | 0.10154 | 0.1019 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 101 | Accept | 0.10154 | 0.072077 | 0.10154 | 0.10186 | tree | MinLeafSize: 5 |
| 102 | Accept | 0.10154 | 0.071287 | 0.10154 | 0.10199 | tree | MinLeafSize: 5 |
| 103 | Accept | 0.10154 | 0.080003 | 0.10154 | 0.10186 | tree | MinLeafSize: 5 |
| 104 | Accept | 0.12 | 0.07462 | 0.10154 | 0.10189 | tree | MinLeafSize: 13 |
| 105 | Accept | 0.10154 | 0.077244 | 0.10154 | 0.10198 | tree | MinLeafSize: 5 |
| 106 | Accept | 0.12 | 0.080302 | 0.10154 | 0.10173 | tree | MinLeafSize: 4 |
| 107 | Accept | 0.10154 | 0.071858 | 0.10154 | 0.10183 | tree | MinLeafSize: 5 |
| 108 | Accept | 0.10154 | 0.076013 | 0.10154 | 0.10166 | tree | MinLeafSize: 5 |
| 109 | Accept | 0.10154 | 0.079106 | 0.10154 | 0.10164 | tree | MinLeafSize: 5 |
| 110 | Accept | 0.10154 | 0.075807 | 0.10154 | 0.10172 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 111 | Accept | 0.10154 | 0.076903 | 0.10154 | 0.10159 | tree | MinLeafSize: 5 |
| 112 | Accept | 0.10154 | 0.077409 | 0.10154 | 0.10163 | tree | MinLeafSize: 5 |
| 113 | Accept | 0.10154 | 0.069569 | 0.10154 | 0.10165 | tree | MinLeafSize: 5 |
| 114 | Accept | 0.12308 | 0.076744 | 0.10154 | 0.10156 | tree | MinLeafSize: 18 |
| 115 | Accept | 0.19551 | 0.097443 | 0.10154 | 0.10156 | svm | BoxConstraint: 1.2977 | | | | | | | | | KernelScale: 33.654 |
| 116 | Accept | 0.26603 | 0.099474 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.082725 | | | | | | | | | KernelScale: 139.98 |
| 117 | Accept | 0.37179 | 0.077589 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.0065156 | | | | | | | | | KernelScale: 559.53 |
| 118 | Accept | 0.15385 | 0.08281 | 0.10154 | 0.10156 | svm | BoxConstraint: 8.5337 | | | | | | | | | KernelScale: 482.07 |
| 119 | Accept | 0.20833 | 0.074271 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.1729 | | | | | | | | | KernelScale: 980.38 |
| 120 | Accept | 0.17949 | 0.081488 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.8028 | | | | | | | | | KernelScale: 578.14 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 121 | Accept | 0.22115 | 0.076901 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.19345 | | | | | | | | | KernelScale: 367.75 |
| 122 | Accept | 0.17308 | 0.076171 | 0.10154 | 0.10156 | svm | BoxConstraint: 10.344 | | | | | | | | | KernelScale: 679.05 |
| 123 | Accept | 0.125 | 0.09187 | 0.10154 | 0.10156 | svm | BoxConstraint: 72.626 | | | | | | | | | KernelScale: 228.42 |
| 124 | Accept | 0.13462 | 0.15746 | 0.10154 | 0.10156 | svm | BoxConstraint: 586.5 | | | | | | | | | KernelScale: 176.02 |
| 125 | Accept | 0.13462 | 0.093366 | 0.10154 | 0.10156 | svm | BoxConstraint: 83.771 | | | | | | | | | KernelScale: 117.21 |
| 126 | Accept | 0.22436 | 0.10124 | 0.10154 | 0.10156 | svm | BoxConstraint: 962.52 | | | | | | | | | KernelScale: 20.898 |
| 127 | Accept | 0.15705 | 0.1133 | 0.10154 | 0.10156 | svm | BoxConstraint: 29.038 | | | | | | | | | KernelScale: 66.563 |
| 128 | Accept | 0.16346 | 0.10388 | 0.10154 | 0.10156 | svm | BoxConstraint: 156.8 | | | | | | | | | KernelScale: 62.775 |
| 129 | Accept | 0.13782 | 0.087555 | 0.10154 | 0.10156 | svm | BoxConstraint: 94.357 | | | | | | | | | KernelScale: 932.27 |
| 130 | Accept | 0.12821 | 0.093318 | 0.10154 | 0.10156 | svm | BoxConstraint: 25.982 | | | | | | | | | KernelScale: 247.06 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 131 | Accept | 0.13462 | 0.08648 | 0.10154 | 0.10156 | svm | BoxConstraint: 16.818 | | | | | | | | | KernelScale: 352.46 |
| 132 | Accept | 0.14103 | 0.090667 | 0.10154 | 0.10156 | svm | BoxConstraint: 15.817 | | | | | | | | | KernelScale: 130.15 |
| 133 | Accept | 0.12179 | 0.089208 | 0.10154 | 0.10156 | svm | BoxConstraint: 74.054 | | | | | | | | | KernelScale: 555.66 |
| 134 | Accept | 0.21474 | 0.082634 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.80097 | | | | | | | | | KernelScale: 239.08 |
| 135 | Accept | 0.125 | 0.088947 | 0.10154 | 0.10156 | svm | BoxConstraint: 47.244 | | | | | | | | | KernelScale: 214.51 |
| 136 | Accept | 0.13141 | 0.07954 | 0.10154 | 0.10156 | svm | BoxConstraint: 13.389 | | | | | | | | | KernelScale: 145.55 |
| 137 | Accept | 0.13782 | 0.091167 | 0.10154 | 0.10156 | svm | BoxConstraint: 60.755 | | | | | | | | | KernelScale: 131.88 |
| 138 | Accept | 0.125 | 0.087338 | 0.10154 | 0.10156 | svm | BoxConstraint: 11.454 | | | | | | | | | KernelScale: 176.64 |
| 139 | Accept | 0.125 | 0.080567 | 0.10154 | 0.10156 | svm | BoxConstraint: 11.85 | | | | | | | | | KernelScale: 160.69 |
| 140 | Accept | 0.16667 | 0.083852 | 0.10154 | 0.10156 | svm | BoxConstraint: 11.155 | | | | | | | | | KernelScale: 652.28 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 141 | Accept | 0.125 | 0.089769 | 0.10154 | 0.10156 | svm | BoxConstraint: 10.933 | | | | | | | | | KernelScale: 172.97 |
| 142 | Accept | 0.21474 | 0.1003 | 0.10154 | 0.10156 | svm | BoxConstraint: 16.172 | | | | | | | | | KernelScale: 22.373 |
| 143 | Accept | 0.13782 | 0.0833 | 0.10154 | 0.10156 | svm | BoxConstraint: 7.6332 | | | | | | | | | KernelScale: 178.51 |
| 144 | Accept | 0.13462 | 0.10562 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.7646 | | | | | | | | | KernelScale: 160.72 |
| 145 | Accept | 0.14103 | 0.076631 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.8617 | | | | | | | | | KernelScale: 189.52 |
| 146 | Accept | 0.15385 | 0.082538 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.0384 | | | | | | | | | KernelScale: 332.89 |
| 147 | Accept | 0.14423 | 0.077025 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.7255 | | | | | | | | | KernelScale: 182.19 |
| 148 | Accept | 0.13782 | 0.089308 | 0.10154 | 0.10156 | svm | BoxConstraint: 70.212 | | | | | | | | | KernelScale: 818.46 |
| 149 | Accept | 0.14103 | 0.08348 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.493 | | | | | | | | | KernelScale: 230.68 |
| 150 | Accept | 0.13782 | 0.079744 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.3564 | | | | | | | | | KernelScale: 111.96 |
__________________________________________________________ Optimization completed. Total iterations: 150 Total elapsed time: 835.2167 seconds Total time for training and validation: 193.4979 seconds Best observed learner is a tree model with: MinLeafSize: 5 Observed validation loss: 0.10154 Time for training and validation: 0.079656 seconds Best estimated learner (returned model) is a tree model with: MinLeafSize: 5 Estimated validation loss: 0.10156 Estimated time for training and validation: 0.076205 seconds Documentation for fitcauto display
The final model returned by fitcauto
corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (carsTrain
), the listed Learner
(or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
Evaluate the performance of the model on the test set.
testAccuracy = 1 - loss(Mdl,carsTest,'Origin')
testAccuracy = 0.9143
confusionchart(carsTest.Origin,predict(Mdl,carsTest))
Use fitcauto
to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in separate variables.
Load Data
Load the humanactivity
data set. This data set contains 24,075 observations of five physical human activities: Sitting (1), Standing (2), Walking (3), Running (4), and Dancing (5). Each observation has 60 features extracted from acceleration data measured by smartphone accelerometer sensors. The variable feat
contains the predictor data matrix of the 60 features for the 24,075 observations, and the response variable actid
contains the activity IDs for the observations as integers.
load humanactivity
Partition Data
Partition the data into training and test sets. Use 90% of the observations to select a model, and 10% of the observations to validate the final model returned by fitcauto
. Use cvpartition
to reserve 10% of the observations for testing.
rng('default') % For reproducibility of the partition c = cvpartition(actid,'Holdout',0.10); trainingIndices = training(c); % Indices for the training set XTrain = feat(trainingIndices,:); YTrain = actid(trainingIndices); testIndices = test(c); % Indices for the test set XTest = feat(testIndices,:); YTest = actid(testIndices);
Run fitcauto
Pass the training data to fitcauto
. By default, fitcauto
determines appropriate model (or learner) types to try, uses Bayesian optimization to find good hyperparameter values for those models, and returns a trained model with the best expected performance. Specify to run the optimization in parallel (requires Parallel Computing Toolbox™). Return a second output OptimizationResults
that contains the details of the Bayesian optimization.
Expect this model selection process to take some time. By default, fitcauto
provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
options = struct('UseParallel',true); [Mdl,OptimizationResults] = fitcauto(XTrain,YTrain,'HyperparameterOptimizationOptions',options);
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Starting parallel pool (parpool) using the 'local' profile ... Connected to the parallel pool (number of workers: 6). Copying objective function to workers... Done copying objective function to workers.
Learner types to explore: ensemble, knn, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 150 Total time (MaxTime): Inf
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 1 | 6 | Best | 0.28088 | 48.752 | 0.28088 | 0.28088 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.22686 | | | | | | | | | | KernelScale: 330.4 |
| 2 | 6 | Best | 0.036459 | 51.455 | 0.036459 | 0.036459 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 254 | | | | | | | | | | MinLeafSize: 1786 | | | | | | | | | | MaxNumSplits: 12 |
| 3 | 6 | Best | 0.025845 | 5.1379 | 0.025845 | 0.025845 | tree | MinLeafSize: 59 |
| 4 | 6 | Best | 0.006415 | 60.999 | 0.006415 | 0.021738 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 214 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 23 |
| 5 | 6 | Accept | 0.025845 | 4.8823 | 0.006415 | 0.021738 | tree | MinLeafSize: 59 |
| 6 | 6 | Accept | 0.017768 | 5.4601 | 0.006415 | 0.021738 | tree | MinLeafSize: 9 |
| 7 | 6 | Accept | 0.050212 | 1.1024 | 0.006415 | 0.021738 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 8 | 6 | Accept | 0.050212 | 0.64183 | 0.006415 | 0.021738 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 9 | 6 | Accept | 0.019568 | 151.32 | 0.006415 | 0.021152 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 218 | | | | | | | | | | MinLeafSize: 2 | | | | | | | | | | MaxNumSplits: 63 |
| 10 | 6 | Accept | 0.026537 | 165.42 | 0.006415 | 0.022035 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 264 | | | | | | | | | | MinLeafSize: 7 | | | | | | | | | | MaxNumSplits: 36 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 11 | 6 | Accept | 0.59166 | 29.107 | 0.006415 | 0.022035 | nb | DistributionNames: kernel | | | | | | | | | | Width: 4.7212e-14 |
| 12 | 6 | Accept | 0.021645 | 50.626 | 0.006415 | 0.022122 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 243 | | | | | | | | | | MinLeafSize: 1247 | | | | | | | | | | MaxNumSplits: 45 |
| 13 | 6 | Accept | 0.043567 | 24.583 | 0.006415 | 0.022122 | knn | NumNeighbors: 144 |
| 14 | 6 | Accept | 0.028844 | 22.503 | 0.006415 | 0.022122 | knn | NumNeighbors: 18 |
| 15 | 6 | Accept | 0.04389 | 157.12 | 0.006415 | 0.022122 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.068467 | | | | | | | | | | KernelScale: 117.01 |
| 16 | 6 | Accept | 0.024598 | 20.721 | 0.006415 | 0.022122 | knn | NumNeighbors: 7 |
| 17 | 6 | Accept | 0.03009 | 20.969 | 0.006415 | 0.022122 | knn | NumNeighbors: 27 |
| 18 | 6 | Accept | 0.016753 | 6.5065 | 0.006415 | 0.021547 | tree | MinLeafSize: 2 |
| 19 | 6 | Accept | 0.040059 | 4.3496 | 0.006415 | 0.022122 | tree | MinLeafSize: 166 |
| 20 | 6 | Accept | 0.060319 | 2.2673 | 0.006415 | 0.022122 | tree | MinLeafSize: 1881 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 21 | 6 | Accept | 0.050212 | 0.97596 | 0.006415 | 0.022122 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 22 | 6 | Accept | 0.036552 | 20.775 | 0.006415 | 0.022122 | knn | NumNeighbors: 67 |
| 23 | 6 | Accept | 0.050212 | 0.55594 | 0.006415 | 0.022122 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 24 | 6 | Accept | 0.11076 | 36.229 | 0.006415 | 0.022122 | knn | NumNeighbors: 2637 |
| 25 | 6 | Accept | 0.27884 | 66.582 | 0.006415 | 0.024532 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 287 | | | | | | | | | | MinLeafSize: 4344 | | | | | | | | | | MaxNumSplits: 48 |
| 26 | 6 | Accept | 0.58127 | 31.861 | 0.006415 | 0.024532 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.6293e-06 |
| 27 | 6 | Accept | 0.01583 | 5.7511 | 0.006415 | 0.020656 | tree | MinLeafSize: 1 |
| 28 | 6 | Accept | 0.069319 | 1.806 | 0.006415 | 0.02077 | tree | MinLeafSize: 2284 |
| 29 | 6 | Accept | 0.59166 | 352.55 | 0.006415 | 0.02077 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 790.4 | | | | | | | | | | KernelScale: 0.014348 |
| 30 | 6 | Accept | 0.043336 | 3.7865 | 0.006415 | 0.020133 | tree | MinLeafSize: 432 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 31 | 6 | Accept | 0.10555 | 34.294 | 0.006415 | 0.020133 | knn | NumNeighbors: 2430 |
| 32 | 6 | Accept | 0.021276 | 4.582 | 0.006415 | 0.018661 | tree | MinLeafSize: 17 |
| 33 | 5 | Accept | 0.030829 | 159.57 | 0.006415 | 0.018642 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 288 | | | | | | | | | | MinLeafSize: 45 | | | | | | | | | | MaxNumSplits: 23 | | 34 | 5 | Accept | 0.014307 | 49.903 | 0.006415 | 0.018642 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | | MinLeafSize: 587 | | | | | | | | | | MaxNumSplits: 11 |
| 35 | 5 | Accept | 0.050212 | 1.025 | 0.006415 | 0.018642 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 36 | 6 | Accept | 0.74165 | 24.131 | 0.006415 | 0.018661 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 217 | | | | | | | | | | MinLeafSize: 8856 | | | | | | | | | | MaxNumSplits: 36 |
| 37 | 6 | Accept | 0.4226 | 558.84 | 0.006415 | 0.018661 | nb | DistributionNames: kernel | | | | | | | | | | Width: 72.906 |
| 38 | 6 | Accept | 0.57615 | 317.96 | 0.006415 | 0.018661 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.2347 | | | | | | | | | | KernelScale: 0.16176 |
| 39 | 6 | Accept | 0.59166 | 26.64 | 0.006415 | 0.018661 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.191e-07 |
| 40 | 6 | Accept | 0.087087 | 30.271 | 0.006415 | 0.018661 | knn | NumNeighbors: 1634 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 41 | 6 | Accept | 0.73985 | 551.47 | 0.006415 | 0.018661 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1055.8 |
| 42 | 4 | Accept | 0.025983 | 146.08 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | | MinLeafSize: 73 | | | | | | | | | | MaxNumSplits: 43 | | 43 | 4 | Accept | 0.02566 | 146.73 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | | MinLeafSize: 73 | | | | | | | | | | MaxNumSplits: 43 | | 44 | 4 | Accept | 0.024922 | 124.65 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 206 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 39 |
| 45 | 6 | Accept | 0.025891 | 23.638 | 0.006415 | 0.018661 | knn | NumNeighbors: 6 |
| 46 | 5 | Accept | 0.025891 | 24.038 | 0.006415 | 0.018661 | knn | NumNeighbors: 6 | | 47 | 5 | Accept | 0.025891 | 23.646 | 0.006415 | 0.018661 | knn | NumNeighbors: 6 |
| 48 | 6 | Accept | 0.03009 | 188.48 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 299 | | | | | | | | | | MinLeafSize: 31 | | | | | | | | | | MaxNumSplits: 25 |
| 49 | 6 | Accept | 0.017214 | 6.0889 | 0.006415 | 0.017935 | tree | MinLeafSize: 4 |
| 50 | 6 | Accept | 0.01726 | 5.6027 | 0.006415 | 0.017303 | tree | MinLeafSize: 5 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 51 | 6 | Accept | 0.037244 | 158.2 | 0.006415 | 0.017303 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 5.9571 | | | | | | | | | | KernelScale: 840.87 |
| 52 | 6 | Accept | 0.046474 | 190.01 | 0.006415 | 0.017303 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 2.9119 | | | | | | | | | | KernelScale: 12.771 |
| 53 | 6 | Accept | 0.032398 | 155.47 | 0.006415 | 0.017303 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 253 | | | | | | | | | | MinLeafSize: 14 | | | | | | | | | | MaxNumSplits: 22 |
| 54 | 6 | Accept | 0.054135 | 3.0156 | 0.006415 | 0.017093 | tree | MinLeafSize: 783 |
| 55 | 6 | Accept | 0.049797 | 28.421 | 0.006415 | 0.017093 | knn | NumNeighbors: 331 |
| 56 | 6 | Accept | 0.046566 | 27.524 | 0.006415 | 0.017093 | knn | NumNeighbors: 193 |
| 57 | 6 | Accept | 0.36307 | 711.57 | 0.006415 | 0.017093 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.0011107 | | | | | | | | | | KernelScale: 0.80966 |
| 58 | 6 | Accept | 0.022706 | 23.417 | 0.006415 | 0.017093 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.3505 | | | | | | | | | | KernelScale: 127.55 |
| 59 | 6 | Accept | 0.028798 | 4.1232 | 0.006415 | 0.01733 | tree | MinLeafSize: 84 |
| 60 | 6 | Accept | 0.041351 | 28.037 | 0.006415 | 0.01733 | knn | NumNeighbors: 124 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 61 | 6 | Accept | 0.030044 | 26.265 | 0.006415 | 0.01733 | knn | NumNeighbors: 26 |
| 62 | 6 | Accept | 0.11838 | 284.42 | 0.006415 | 0.01733 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.0027874 | | | | | | | | | | KernelScale: 33.944 |
| 63 | 6 | Accept | 0.47116 | 54.038 | 0.006415 | 0.01733 | nb | DistributionNames: kernel | | | | | | | | | | Width: 4.7553e-05 |
| 64 | 6 | Accept | 0.26574 | 104.86 | 0.006415 | 0.01733 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0011441 |
| 65 | 6 | Accept | 0.050212 | 0.72243 | 0.006415 | 0.01733 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 66 | 6 | Accept | 0.01726 | 5.6362 | 0.006415 | 0.016846 | tree | MinLeafSize: 5 |
| 67 | 6 | Accept | 0.077072 | 268.17 | 0.006415 | 0.016846 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.026902 |
| 68 | 6 | Accept | 0.031613 | 25.909 | 0.006415 | 0.016846 | knn | NumNeighbors: 33 |
| 69 | 6 | Accept | 0.02003 | 92.379 | 0.006415 | 0.016846 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 609.47 | | | | | | | | | | KernelScale: 39.88 |
| 70 | 6 | Accept | 0.1145 | 92.795 | 0.006415 | 0.016846 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 292 | | | | | | | | | | MinLeafSize: 3870 | | | | | | | | | | MaxNumSplits: 33 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 71 | 6 | Accept | 0.012968 | 53.299 | 0.006415 | 0.016846 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 208 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 10 |
| 72 | 6 | Accept | 0.011999 | 53.54 | 0.006415 | 0.016846 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 215 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 11 |
| 73 | 6 | Accept | 0.011999 | 50.887 | 0.006415 | 0.012076 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 205 | | | | | | | | | | MinLeafSize: 2 | | | | | | | | | | MaxNumSplits: 11 |
| 74 | 6 | Accept | 0.039921 | 40.808 | 0.006415 | 0.012733 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 200 | | | | | | | | | | MinLeafSize: 1921 | | | | | | | | | | MaxNumSplits: 11 |
| 75 | 6 | Accept | 0.04232 | 98.684 | 0.006415 | 0.012754 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 220 | | | | | | | | | | MinLeafSize: 1495 | | | | | | | | | | MaxNumSplits: 11 |
| 76 | 6 | Accept | 0.094702 | 397.88 | 0.006415 | 0.012754 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 671.76 | | | | | | | | | | KernelScale: 4.6418 |
| 77 | 6 | Accept | 0.065165 | 44.801 | 0.006415 | 0.013179 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 233 | | | | | | | | | | MinLeafSize: 2586 | | | | | | | | | | MaxNumSplits: 35 |
| 78 | 6 | Accept | 0.22503 | 164.64 | 0.006415 | 0.013179 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 57.71 | | | | | | | | | | KernelScale: 2.0632 |
| 79 | 6 | Accept | 0.03069 | 44.42 | 0.006415 | 0.013027 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 213 | | | | | | | | | | MinLeafSize: 1619 | | | | | | | | | | MaxNumSplits: 95 |
| 80 | 6 | Accept | 0.11459 | 71.56 | 0.006415 | 0.012064 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 212 | | | | | | | | | | MinLeafSize: 2669 | | | | | | | | | | MaxNumSplits: 31 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 81 | 6 | Accept | 0.044582 | 41.462 | 0.006415 | 0.013176 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 207 | | | | | | | | | | MinLeafSize: 2211 | | | | | | | | | | MaxNumSplits: 56 |
| 82 | 6 | Accept | 0.014722 | 223.7 | 0.006415 | 0.013176 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 832.84 | | | | | | | | | | KernelScale: 392.65 |
| 83 | 6 | Accept | 0.018322 | 23.34 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 322.92 | | | | | | | | | | KernelScale: 542.76 |
| 84 | 6 | Accept | 0.016984 | 28.833 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 727.84 | | | | | | | | | | KernelScale: 558.25 |
| 85 | 6 | Accept | 0.59166 | 2135.3 | 0.006415 | 0.013176 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.045413 | | | | | | | | | | KernelScale: 0.0034709 |
| 86 | 6 | Accept | 0.012461 | 39.005 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 806.52 | | | | | | | | | | KernelScale: 244.64 |
| 87 | 6 | Accept | 0.016753 | 22.649 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 173.38 | | | | | | | | | | KernelScale: 329.41 |
| 88 | 6 | Accept | 0.016845 | 28.292 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 242.37 | | | | | | | | | | KernelScale: 62.644 |
| 89 | 6 | Accept | 0.016799 | 20.771 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 80.674 | | | | | | | | | | KernelScale: 242.97 |
| 90 | 6 | Accept | 0.041259 | 175.27 | 0.006415 | 0.013176 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 2.0147 | | | | | | | | | | KernelScale: 549.95 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 91 | 6 | Accept | 0.017953 | 19.778 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 63.367 | | | | | | | | | | KernelScale: 263.72 |
| 92 | 6 | Accept | 0.019568 | 19.245 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 55.612 | | | | | | | | | | KernelScale: 344.11 |
| 93 | 6 | Accept | 0.016061 | 19.511 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 60.019 | | | | | | | | | | KernelScale: 185.49 |
| 94 | 6 | Accept | 0.11847 | 149.5 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 770.37 | | | | | | | | | | KernelScale: 3.1188 |
| 95 | 6 | Accept | 0.017953 | 18.428 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 43.94 | | | | | | | | | | KernelScale: 234.89 |
| 96 | 6 | Accept | 0.025106 | 25.204 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.1294 | | | | | | | | | | KernelScale: 237.71 |
| 97 | 6 | Accept | 0.011676 | 70.358 | 0.006415 | 0.012446 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 283 | | | | | | | | | | MinLeafSize: 4 | | | | | | | | | | MaxNumSplits: 10 |
| 98 | 6 | Accept | 0.031983 | 37.179 | 0.006415 | 0.012446 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.832 | | | | | | | | | | KernelScale: 449.73 |
| 99 | 6 | Accept | 0.0097379 | 76.639 | 0.006415 | 0.011402 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 299 | | | | | | | | | | MinLeafSize: 4 | | | | | | | | | | MaxNumSplits: 12 |
| 100 | 6 | Accept | 0.22416 | 684.65 | 0.006415 | 0.011402 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 632.44 | | | | | | | | | | KernelScale: 1.8647 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 101 | 6 | Accept | 0.11478 | 87.263 | 0.006415 | 0.011913 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 272 | | | | | | | | | | MinLeafSize: 3176 | | | | | | | | | | MaxNumSplits: 54 |
| 102 | 6 | Accept | 0.0081687 | 75.177 | 0.006415 | 0.01045 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 291 | | | | | | | | | | MinLeafSize: 6 | | | | | | | | | | MaxNumSplits: 14 |
| 103 | 6 | Accept | 0.010753 | 70.01 | 0.006415 | 0.010258 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 284 | | | | | | | | | | MinLeafSize: 1 | | | | | | | | | | MaxNumSplits: 11 |
| 104 | 6 | Accept | 0.36076 | 77 | 0.006415 | 0.010258 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0054235 | | | | | | | | | | KernelScale: 149.67 |
| 105 | 6 | Accept | 0.0084456 | 62.748 | 0.006415 | 0.0092051 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 214 | | | | | | | | | | MinLeafSize: 3 | | | | | | | | | | MaxNumSplits: 16 |
| 106 | 6 | Accept | 0.012738 | 32.34 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 529.51 | | | | | | | | | | KernelScale: 161.83 |
| 107 | 6 | Accept | 0.031521 | 41.482 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.6738 | | | | | | | | | | KernelScale: 289.2 |
| 108 | 6 | Accept | 0.021368 | 38.604 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 567.35 | | | | | | | | | | KernelScale: 36.573 |
| 109 | 6 | Accept | 0.050212 | 0.59557 | 0.006415 | 0.0092051 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 110 | 6 | Accept | 0.57883 | 127.15 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0011506 | | | | | | | | | | KernelScale: 211.98 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 111 | 6 | Accept | 0.050212 | 0.685 | 0.006415 | 0.0092051 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 112 | 6 | Accept | 0.029583 | 33.504 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.49417 | | | | | | | | | | KernelScale: 209.72 |
| 113 | 6 | Accept | 0.012138 | 41.159 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 967.85 | | | | | | | | | | KernelScale: 153.85 |
| 114 | 6 | Accept | 0.030921 | 38.482 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.57771 | | | | | | | | | | KernelScale: 262.15 |
| 115 | 6 | Accept | 0.1295 | 52.128 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.35331 | | | | | | | | | | KernelScale: 329.77 |
| 116 | 6 | Accept | 0.028475 | 33.553 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.37306 | | | | | | | | | | KernelScale: 157.01 |
| 117 | 6 | Accept | 0.050212 | 0.62316 | 0.006415 | 0.0092051 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 118 | 6 | Accept | 0.030737 | 37 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.37996 | | | | | | | | | | KernelScale: 204.24 |
| 119 | 6 | Accept | 0.021229 | 36.97 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 803.51 | | | | | | | | | | KernelScale: 38.271 |
| 120 | 6 | Accept | 0.015184 | 24.004 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 119.72 | | | | | | | | | | KernelScale: 145.99 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 121 | 6 | Accept | 0.06955 | 101.38 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 39.152 | | | | | | | | | | KernelScale: 8.5572 |
| 122 | 6 | Accept | 0.028291 | 32.637 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.32264 | | | | | | | | | | KernelScale: 134.73 |
| 123 | 6 | Accept | 0.041951 | 51.65 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.28203 | | | | | | | | | | KernelScale: 237.1 |
| 124 | 6 | Accept | 0.039921 | 216.38 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.049329 | | | | | | | | | | KernelScale: 190.24 |
| 125 | 6 | Accept | 0.048828 | 91.888 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 180.12 | | | | | | | | | | KernelScale: 12.235 |
| 126 | 6 | Accept | 0.030644 | 38.563 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.25725 | | | | | | | | | | KernelScale: 171.19 |
| 127 | 6 | Accept | 0.029537 | 37.409 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.25269 | | | | | | | | | | KernelScale: 145.85 |
| 128 | 6 | Accept | 0.02949 | 31.222 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.24034 | | | | | | | | | | KernelScale: 59.755 |
| 129 | 6 | Accept | 0.029537 | 130.19 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 76.533 | | | | | | | | | | KernelScale: 22.982 |
| 130 | 6 | Accept | 0.094702 | 174.14 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.043666 | | | | | | | | | | KernelScale: 25.411 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 131 | 6 | Accept | 0.037382 | 46.207 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.23867 | | | | | | | | | | KernelScale: 204.62 |
| 132 | 6 | Accept | 0.022152 | 40.588 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 887.62 | | | | | | | | | | KernelScale: 32.987 |
| 133 | 6 | Accept | 0.032629 | 32.651 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.22511 | | | | | | | | | | KernelScale: 52.294 |
| 134 | 6 | Accept | 0.02806 | 131.12 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.20346 | | | | | | | | | | KernelScale: 122.85 |
| 135 | 6 | Accept | 0.59166 | 3480 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 658.37 | | | | | | | | | | KernelScale: 0.0016161 |
| 136 | 6 | Accept | 0.59166 | 3481.3 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 658.37 | | | | | | | | | | KernelScale: 0.0016161 |
| 137 | 6 | Accept | 0.032121 | 44.172 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.20648 | | | | | | | | | | KernelScale: 162.33 |
| 138 | 6 | Accept | 0.30755 | 71.9 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.015886 | | | | | | | | | | KernelScale: 120.23 |
| 139 | 6 | Accept | 0.040936 | 233.89 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.046614 | | | | | | | | | | KernelScale: 213.78 |
| 140 | 5 | Accept | 0.040521 | 224.72 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.043998 | | | | | | | | | | KernelScale: 195.58 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 141 | 5 | Accept | 0.22208 | 64.659 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.049755 | | | | | | | | | | KernelScale: 151.01 |
| 142 | 6 | Accept | 0.029444 | 30.014 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.21789 | | | | | | | | | | KernelScale: 64.023 |
| 143 | 6 | Accept | 0.046013 | 49.025 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.14805 | | | | | | | | | | KernelScale: 178.98 |
| 144 | 6 | Accept | 0.013291 | 42.726 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 960.14 | | | | | | | | | | KernelScale: 334.83 |
| 145 | 6 | Accept | 0.59166 | 3493.5 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 17.982 | | | | | | | | | | KernelScale: 0.0043756 |
| 146 | 6 | Accept | 0.089533 | 189.71 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.2326 | | | | | | | | | | KernelScale: 15.411 |
| 147 | 6 | Accept | 0.03369 | 49.586 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.21482 | | | | | | | | | | KernelScale: 180.5 |
| 148 | 6 | Accept | 0.013384 | 42.876 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 920.16 | | | | | | | | | | KernelScale: 334.75 |
| 149 | 6 | Accept | 0.027368 | 117.6 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.22783 | | | | | | | | | | KernelScale: 105.59 |
| 150 | 6 | Accept | 0.041674 | 234.61 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.032335 | | | | | | | | | | KernelScale: 182.17 |
__________________________________________________________ Optimization completed. Total iterations: 150 Total elapsed time: 4534.2478 seconds Total time for training and validation: 24883.8563 seconds Best observed learner is an ensemble model with: Method: AdaBoostM2 NumLearningCycles: 214 MinLeafSize: 5 MaxNumSplits: 23 Observed validation loss: 0.006415 Time for training and validation: 60.9987 seconds Best estimated learner (returned model) is an ensemble model with: Method: AdaBoostM2 NumLearningCycles: 214 MinLeafSize: 3 MaxNumSplits: 16 Estimated validation loss: 0.0092051 Estimated time for training and validation: 57.8146 seconds Documentation for fitcauto display
The final model returned by fitcauto
corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (XTrain
and YTrain
), the listed Learner
(or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
Evaluate the final model performance on the test data set.
testAccuracy = 1 - loss(Mdl,XTest,YTest)
testAccuracy = 0.9917
The final model correctly classifies over 99% of the observations.
Use fitcauto
to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in a table. Before passing data to fitcauto
, perform feature selection to remove unimportant predictors from the data set.
Load and Partition Data
Read the sample file CreditRating_Historical.dat
into a table. The predictor data consists of financial ratios and industry sector information for a list of corporate customers. The response variable consists of credit ratings assigned by a rating agency. Preview the first few rows of the data set.
creditrating = readtable('CreditRating_Historical.dat');
head(creditrating)
ans=8×8 table
ID WC_TA RE_TA EBIT_TA MVE_BVTD S_TA Industry Rating
_____ ______ ______ _______ ________ _____ ________ _______
62394 0.013 0.104 0.036 0.447 0.142 3 {'BB' }
48608 0.232 0.335 0.062 1.969 0.281 8 {'A' }
42444 0.311 0.367 0.074 1.935 0.366 1 {'A' }
48631 0.194 0.263 0.062 1.017 0.228 4 {'BBB'}
43768 0.121 0.413 0.057 3.647 0.466 12 {'AAA'}
39255 -0.117 -0.799 0.01 0.179 0.082 4 {'CCC'}
62236 0.087 0.158 0.049 0.816 0.324 2 {'BBB'}
39354 0.005 0.181 0.034 2.597 0.388 7 {'AA' }
Because each value in the ID
variable is a unique customer ID, that is, length(unique(creditrating.ID))
is equal to the number of observations in creditrating
, the ID
variable is a poor predictor. Remove the ID
variable from the table, and convert the Industry
variable to a categorical
variable.
creditrating = removevars(creditrating,'ID');
creditrating.Industry = categorical(creditrating.Industry);
Partition the data into training and test sets. Use approximately 85% of the observations for the model selection and hyperparameter tuning process, and 15% of the observations to test the performance of the final model returned by fitcauto
on new data. Use cvpartition
to partition the data.
rng('default') % For reproducibility of the partition c = cvpartition(creditrating.Rating,'Holdout',0.15); trainingIndices = training(c); % Indices for the training set testIndices = test(c); % Indices for the test set creditTrain = creditrating(trainingIndices,:); creditTest = creditrating(testIndices,:);
Perform Feature Selection
Before passing the training data to fitcauto
, find the important predictors by using the fscchi2
function. Visualize the predictor scores by using the bar
function. Because some scores can be Inf
, and bar
discards Inf
values, plot the finite scores first and then plot a finite representation of the Inf
scores in a different color.
[idx,scores] = fscchi2(creditTrain,'Rating'); bar(scores(idx)) % Represents finite scores hold on veryImportant = isinf(scores); finiteMax = max(scores(~veryImportant)); bar(finiteMax*veryImportant(idx)) % Represents Inf scores hold off xticklabels(strrep(creditTrain.Properties.VariableNames(idx),'_','\_')) xtickangle(45) legend({'Finite Scores','Inf Scores'})
Notice that the Industry
predictor has a low score corresponding to a p-value that is greater than 0.05, which indicates that Industry
might not be an important feature. Remove the Industry
feature from the training and test data sets.
creditTrain = removevars(creditTrain,'Industry'); creditTest = removevars(creditTest,'Industry');
Run fitcauto
Pass the training data to fitcauto
. The function uses Bayesian optimization to select models and their hyperparameter values, and returns a trained model Mdl
with the best expected performance. Specify to try all available learner types and run the optimization in parallel (requires Parallel Computing Toolbox™). Return a second output Results
that contains the details of the Bayesian optimization.
Expect this process to take some time. By default, fitcauto
provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
options = struct('UseParallel',true); [Mdl,Results] = fitcauto(creditTrain,'Rating', ... 'Learners','all','HyperparameterOptimizationOptions',options);
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Starting parallel pool (parpool) using the 'local' profile ... Connected to the parallel pool (number of workers: 6).
Copying objective function to workers... Done copying objective function to workers.
Learner types to explore: discr, ensemble, kernel, knn, linear, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 240 Total time (MaxTime): Inf
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 1 | 6 | Best | 0.42716 | 3.0379 | 0.42716 | 0.42716 | discr | Delta: 0.00046441 | | | | | | | | | | Gamma: 0.2485 |
| 2 | 4 | Accept | 0.74185 | 4.7899 | 0.24948 | 0.29794 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.48455 | | | | | | | | | | KernelScale: 354.44 | | 3 | 4 | Best | 0.24948 | 5.0813 | 0.24948 | 0.29794 | linear | Coding: onevsone | | | | | | | | | | Lambda: 6.3551e-08 | | | | | | | | | | Learner: logistic | | 4 | 4 | Accept | 0.29794 | 3.7295 | 0.24948 | 0.29794 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 12 | | | | | | | | | | LearnRate: 0.063776 | | | | | | | | | | MinLeafSize: 277 |
| 5 | 3 | Accept | 0.25097 | 9.2655 | 0.24948 | 0.25067 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 7.8433 | | | | | | | | | | Lambda: 1.4468e-06 | | 6 | 3 | Accept | 0.25067 | 0.81139 | 0.24948 | 0.25067 | knn | NumNeighbors: 105 | | | | | | | | | | Distance: minkowski |
| 7 | 6 | Accept | 0.52917 | 2.3362 | 0.24948 | 0.25067 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.002417 | | | | | | | | | | KernelScale: 356.9 |
| 8 | 3 | Accept | 0.55818 | 0.63908 | 0.24948 | 0.25067 | discr | Delta: 0.98612 | | | | | | | | | | Gamma: 0.86519 | | 9 | 3 | Accept | 0.3781 | 1.6777 | 0.24948 | 0.25067 | linear | Coding: onevsall | | | | | | | | | | Lambda: 1.0412e-06 | | | | | | | | | | Learner: logistic | | 10 | 3 | Accept | 0.43225 | 0.80766 | 0.24948 | 0.25067 | discr | Delta: 0.00013711 | | | | | | | | | | Gamma: 0.60585 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 11 | 3 | Accept | 0.47712 | 3.3756 | 0.24948 | 0.25067 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 2.7347 | | | | | | | | | | KernelScale: 24.465 |
| 12 | 6 | Accept | 0.25695 | 2.3709 | 0.24948 | 0.25067 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.057566 |
| 13 | 4 | Accept | 0.26413 | 0.49941 | 0.24379 | 0.25067 | tree | MinLeafSize: 30 | | 14 | 4 | Accept | 0.42327 | 0.85101 | 0.24379 | 0.25067 | knn | NumNeighbors: 56 | | | | | | | | | | Distance: cosine | | 15 | 4 | Best | 0.24379 | 1.8084 | 0.24379 | 0.25067 | linear | Coding: onevsone | | | | | | | | | | Lambda: 5.9172e-05 | | | | | | | | | | Learner: svm |
| 16 | 3 | Accept | 0.81544 | 4.9586 | 0.24379 | 0.25067 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0043375 | | | | | | | | | | Lambda: 0.0023789 | | 17 | 3 | Accept | 0.45169 | 0.96723 | 0.24379 | 0.25067 | linear | Coding: onevsall | | | | | | | | | | Lambda: 0.0028505 | | | | | | | | | | Learner: svm |
| 18 | 6 | Accept | 0.33712 | 0.19972 | 0.24379 | 0.25695 | knn | NumNeighbors: 1 | | | | | | | | | | Distance: cityblock |
| 19 | 3 | Accept | 0.4834 | 0.38951 | 0.24379 | 0.25695 | knn | NumNeighbors: 72 | | | | | | | | | | Distance: correlation | | 20 | 3 | Accept | 0.46336 | 0.78881 | 0.24379 | 0.25695 | linear | Coding: onevsall | | | | | | | | | | Lambda: 0.0075732 | | | | | | | | | | Learner: svm | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 21 | 3 | Accept | 0.82082 | 2.9223 | 0.24379 | 0.25695 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0042587 | | | | | | | | | | Lambda: 0.0014754 | | 22 | 3 | Accept | 0.61292 | 1.8557 | 0.24379 | 0.25695 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 13 | | | | | | | | | | LearnRate: 0.055349 | | | | | | | | | | MinLeafSize: 910 |
| 23 | 6 | Accept | 0.43255 | 0.68689 | 0.24379 | 0.25695 | discr | Delta: 0.016844 | | | | | | | | | | Gamma: 0.64466 |
| 24 | 4 | Accept | 0.28866 | 1.9017 | 0.24379 | 0.25695 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 10 | | | | | | | | | | LearnRate: 0.11662 | | | | | | | | | | MinLeafSize: 181 | | 25 | 4 | Accept | 0.74185 | 1.3546 | 0.24379 | 0.25695 | knn | NumNeighbors: 1314 | | | | | | | | | | Distance: hamming | | 26 | 4 | Accept | 0.42746 | 0.69002 | 0.24379 | 0.25695 | discr | Delta: 2.2544e-06 | | | | | | | | | | Gamma: 0.87275 |
| 27 | 3 | Accept | 0.25606 | 12.143 | 0.24379 | 0.25498 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 132 | | | | | | | | | | LearnRate: 0.92674 | | | | | | | | | | MinLeafSize: 127 | | 28 | 3 | Accept | 0.25366 | 2.4732 | 0.24379 | 0.25498 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.10033 |
| 29 | 6 | Accept | 0.66796 | 0.22938 | 0.24379 | 0.25498 | knn | NumNeighbors: 77 | | | | | | | | | | Distance: jaccard |
| 30 | 4 | Accept | 0.69488 | 1.9352 | 0.242 | 0.25498 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 8.4886 | | | | | | | | | | KernelScale: 192.19 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 31 | 4 | Best | 0.242 | 1.9467 | 0.242 | 0.25498 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 5.425 | | | | | | | | | | KernelScale: 2.434 | | 32 | 4 | Accept | 0.32306 | 0.48104 | 0.242 | 0.25498 | tree | MinLeafSize: 2 |
| 33 | 4 | Accept | 0.43225 | 0.10463 | 0.242 | 0.25498 | discr | Delta: 0.00015292 | | | | | | | | | | Gamma: 0.51045 |
| 34 | 4 | Accept | 0.32994 | 0.2065 | 0.242 | 0.25498 | tree | MinLeafSize: 3 |
| 35 | 6 | Accept | 0.53814 | 2.8748 | 0.242 | 0.25498 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.8148 | | | | | | | | | | KernelScale: 382.11 |
| 36 | 4 | Accept | 0.24529 | 105.62 | 0.242 | 0.25498 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.25488 | | | | | | | | | | KernelScale: 0.0037823 | | 37 | 4 | Accept | 0.53814 | 3.3272 | 0.242 | 0.25498 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.8148 | | | | | | | | | | KernelScale: 382.11 | | 38 | 4 | Accept | 0.53814 | 3.8593 | 0.242 | 0.25498 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.8148 | | | | | | | | | | KernelScale: 382.11 |
| 39 | 4 | Accept | 0.25965 | 14.211 | 0.242 | 0.25498 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 150 | | | | | | | | | | LearnRate: 0.014842 | | | | | | | | | | MinLeafSize: 21 |
| 40 | 5 | Accept | 0.42656 | 0.16731 | 0.242 | 0.25498 | discr | Delta: 0.0020866 | | | | | | | | | | Gamma: 0.091054 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 41 | 5 | Accept | 0.42656 | 0.11476 | 0.242 | 0.25498 | discr | Delta: 0.0020866 | | | | | | | | | | Gamma: 0.091054 |
| 42 | 4 | Accept | 0.2767 | 21.389 | 0.242 | 0.25498 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 221 | | | | | | | | | | LearnRate: 0.0028588 | | | | | | | | | | MinLeafSize: 1 | | 43 | 4 | Accept | 0.29973 | 0.1848 | 0.242 | 0.25498 | tree | MinLeafSize: 7 |
| 44 | 4 | Accept | 0.25935 | 20.084 | 0.242 | 0.25498 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 304 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 100 |
| 45 | 3 | Accept | 0.24499 | 6.5071 | 0.242 | 0.26328 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.019387 | | | | | | | | | | KernelScale: 0.0047515 | | 46 | 3 | Accept | 0.28059 | 0.19378 | 0.242 | 0.26328 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 47 | 6 | Accept | 0.27281 | 0.13181 | 0.242 | 0.26328 | knn | NumNeighbors: 8 | | | | | | | | | | Distance: chebychev |
| 48 | 3 | Accept | 0.2429 | 1.5474 | 0.242 | 0.26328 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.16719 | | | | | | | | | | KernelScale: 0.11257 | | 49 | 3 | Accept | 0.2423 | 1.5219 | 0.242 | 0.26328 | linear | Coding: onevsone | | | | | | | | | | Lambda: 4.28e-07 | | | | | | | | | | Learner: svm | | 50 | 3 | Accept | 0.67125 | 0.68464 | 0.242 | 0.26328 | knn | NumNeighbors: 86 | | | | | | | | | | Distance: jaccard | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 51 | 3 | Accept | 0.46485 | 0.8417 | 0.242 | 0.26328 | tree | MinLeafSize: 645 |
| 52 | 6 | Accept | 0.63147 | 3.1926 | 0.242 | 0.26328 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 176.2 | | | | | | | | | | Lambda: 2.3903e-06 |
| 53 | 4 | Accept | 0.37571 | 0.11086 | 0.242 | 0.26706 | tree | MinLeafSize: 473 | | 54 | 4 | Accept | 0.29136 | 0.43908 | 0.242 | 0.26706 | knn | NumNeighbors: 354 | | | | | | | | | | Distance: euclidean | | 55 | 4 | Accept | 0.28059 | 0.50642 | 0.242 | 0.26706 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 56 | 3 | Accept | 0.36375 | 5.2617 | 0.242 | 0.26706 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 28.598 | | | | | | | | | | Lambda: 8.3238e-05 | | 57 | 3 | Accept | 0.27251 | 0.13425 | 0.242 | 0.26706 | tree | MinLeafSize: 20 |
| 58 | 6 | Accept | 0.43225 | 0.083255 | 0.242 | 0.26706 | discr | Delta: 0.021467 | | | | | | | | | | Gamma: 0.66016 |
| 59 | 3 | Accept | 0.28059 | 0.10921 | 0.242 | 0.26106 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | 60 | 3 | Accept | 0.42537 | 0.15885 | 0.242 | 0.26106 | discr | Delta: 0.001728 | | | | | | | | | | Gamma: 0.89471 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 61 | 3 | Accept | 0.81484 | 2.9209 | 0.242 | 0.26106 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0063987 | | | | | | | | | | Lambda: 0.0075855 | | 62 | 3 | Accept | 0.24948 | 2.4493 | 0.242 | 0.26106 | linear | Coding: onevsone | | | | | | | | | | Lambda: 1.3237e-07 | | | | | | | | | | Learner: logistic |
| 63 | 6 | Accept | 0.68531 | 0.40944 | 0.242 | 0.26106 | knn | NumNeighbors: 260 | | | | | | | | | | Distance: jaccard |
| 64 | 3 | Accept | 0.32426 | 9.8071 | 0.242 | 0.26007 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 132 | | | | | | | | | | LearnRate: 0.0014516 | | | | | | | | | | MinLeafSize: 104 | | 65 | 3 | Accept | 0.55369 | 0.69919 | 0.242 | 0.26007 | knn | NumNeighbors: 615 | | | | | | | | | | Distance: correlation | | 66 | 3 | Accept | 0.24319 | 1.5742 | 0.242 | 0.26007 | linear | Coding: onevsone | | | | | | | | | | Lambda: 4.3001e-09 | | | | | | | | | | Learner: svm | | 67 | 3 | Accept | 0.70894 | 1.9822 | 0.242 | 0.26007 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45564 | | | | | | | | | | KernelScale: 51.195 |
| 68 | 6 | Accept | 0.46485 | 0.17082 | 0.242 | 0.26007 | tree | MinLeafSize: 611 |
| 69 | 5 | Accept | 0.74185 | 2.1172 | 0.242 | 0.26007 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.94197 | | | | | | | | | | KernelScale: 524.56 | | 70 | 5 | Accept | 0.83548 | 6.3585 | 0.242 | 0.26007 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0038762 | | | | | | | | | | Lambda: 2.288e-06 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 71 | 3 | Accept | 0.25905 | 16.096 | 0.242 | 0.26007 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 160 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 2 | | 72 | 3 | Accept | 0.28448 | 13.826 | 0.242 | 0.26007 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 161 | | | | | | | | | | LearnRate: 0.7581 | | | | | | | | | | MinLeafSize: 14 | | 73 | 3 | Accept | 0.25695 | 0.24642 | 0.242 | 0.26007 | knn | NumNeighbors: 14 | | | | | | | | | | Distance: cityblock |
| 74 | 6 | Accept | 0.32396 | 1.8799 | 0.242 | 0.26007 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.86904 | | | | | | | | | | Lambda: 0.29724 |
| 75 | 4 | Accept | 0.32456 | 0.20481 | 0.242 | 0.26007 | tree | MinLeafSize: 5 | | 76 | 4 | Accept | 0.32994 | 0.35799 | 0.242 | 0.26007 | tree | MinLeafSize: 3 | | 77 | 4 | Accept | 0.26054 | 4.1734 | 0.242 | 0.26007 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 32 | | | | | | | | | | LearnRate: 0.06853 | | | | | | | | | | MinLeafSize: 19 |
| 78 | 4 | Accept | 0.43703 | 0.92964 | 0.242 | 0.24831 | linear | Coding: onevsall | | | | | | | | | | Lambda: 0.013265 | | | | | | | | | | Learner: logistic |
| 79 | 4 | Accept | 0.31588 | 17.116 | 0.242 | 0.24831 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 201 | | | | | | | | | | LearnRate: 0.0012955 | | | | | | | | | | MinLeafSize: 319 |
| 80 | 3 | Accept | 0.25277 | 7.8173 | 0.242 | 0.24831 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.78697 | | | | | | | | | | Lambda: 4.1197e-06 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 81 | 3 | Accept | 0.43015 | 0.10239 | 0.242 | 0.24831 | discr | Delta: 0.0069822 | | | | | | | | | | Gamma: 0.49526 |
| 82 | 6 | Accept | 0.42208 | 0.096953 | 0.242 | 0.24831 | discr | Delta: 0.057485 | | | | | | | | | | Gamma: 0.045714 |
| 83 | 3 | Accept | 0.52617 | 2.3915 | 0.242 | 0.24831 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.26869 | | | | | | | | | | KernelScale: 17.595 | | 84 | 3 | Accept | 0.43344 | 1.2875 | 0.242 | 0.24831 | knn | NumNeighbors: 119 | | | | | | | | | | Distance: mahalanobis | | 85 | 3 | Accept | 0.30093 | 1.3003 | 0.242 | 0.24831 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.047624 | | | | | | | | | | Learner: svm | | 86 | 3 | Accept | 0.42267 | 0.64506 | 0.242 | 0.24831 | knn | NumNeighbors: 48 | | | | | | | | | | Distance: cosine |
| 87 | 6 | Accept | 0.32905 | 0.26891 | 0.242 | 0.24831 | knn | NumNeighbors: 65 | | | | | | | | | | Distance: seuclidean |
| 88 | 4 | Accept | 0.24349 | 1.9282 | 0.242 | 0.24684 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0024196 | | | | | | | | | | KernelScale: 0.0082547 | | 89 | 4 | Accept | 0.24499 | 1.6543 | 0.242 | 0.24684 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.3697e-06 | | | | | | | | | | Learner: svm | | 90 | 4 | Accept | 0.24469 | 2.1463 | 0.242 | 0.24684 | linear | Coding: onevsone | | | | | | | | | | Lambda: 6.5777e-05 | | | | | | | | | | Learner: logistic |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 91 | 4 | Accept | 0.28059 | 0.15481 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 92 | 5 | Accept | 0.28059 | 0.38138 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | 93 | 5 | Accept | 0.28059 | 0.15804 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 94 | 4 | Accept | 0.29674 | 11.234 | 0.242 | 0.24684 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 147 | | | | | | | | | | LearnRate: 0.95321 | | | | | | | | | | MinLeafSize: 112 | | 95 | 4 | Accept | 0.28059 | 0.12279 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 96 | 4 | Accept | 0.29704 | 1.1016 | 0.242 | 0.24684 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 11 | | | | | | | | | | LearnRate: 0.0072731 | | | | | | | | | | MinLeafSize: 12 |
| 97 | 4 | Accept | 0.28059 | 0.099833 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 98 | 4 | Best | 0.2411 | 1.3634 | 0.2411 | 0.24471 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.00056045 | | | | | | | | | | Learner: svm |
| 99 | 4 | Accept | 0.74185 | 0.10441 | 0.2411 | 0.24471 | discr | Delta: 39.281 | | | | | | | | | | Gamma: 0.77032 |
| 100 | 4 | Accept | 0.30093 | 0.11043 | 0.2411 | 0.24471 | tree | MinLeafSize: 135 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 101 | 4 | Accept | 0.46844 | 0.17182 | 0.2411 | 0.24471 | knn | NumNeighbors: 2 | | | | | | | | | | Distance: cosine |
| 102 | 4 | Accept | 0.25426 | 3.1634 | 0.2411 | 0.24471 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 38 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 28 |
| 103 | 4 | Accept | 0.24469 | 1.6558 | 0.2411 | 0.24558 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.0012413 | | | | | | | | | | Learner: logistic |
| 104 | 4 | Accept | 0.43823 | 3.6572 | 0.2411 | 0.24558 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 13.293 | | | | | | | | | | Lambda: 0.0031304 |
| 105 | 4 | Accept | 0.31977 | 1.3423 | 0.2411 | 0.24556 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.040102 | | | | | | | | | | Learner: logistic |
| 106 | 6 | Accept | 0.24678 | 2.0849 | 0.2411 | 0.24529 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.9918e-05 | | | | | | | | | | Learner: logistic |
| 107 | 5 | Accept | 0.24678 | 1.9627 | 0.2411 | 0.24441 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.9918e-05 | | | | | | | | | | Learner: logistic | | 108 | 5 | Accept | 0.24678 | 2.3413 | 0.2411 | 0.24441 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.9918e-05 | | | | | | | | | | Learner: logistic |
| 109 | 4 | Accept | 0.29435 | 14.698 | 0.2411 | 0.24441 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 249 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 292 | | 110 | 4 | Accept | 0.37152 | 0.098701 | 0.2411 | 0.24441 | tree | MinLeafSize: 375 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 111 | 4 | Accept | 0.25666 | 5.7386 | 0.2411 | 0.24441 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 61 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 3 | | 112 | 4 | Accept | 0.28059 | 0.10645 | 0.2411 | 0.24441 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 113 | 4 | Accept | 0.28059 | 0.10515 | 0.2411 | 0.24441 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 114 | 6 | Accept | 0.74185 | 2.7447 | 0.2411 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 74.975 |
| 115 | 4 | Accept | 0.78552 | 10.495 | 0.2411 | 0.24441 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.0050713 | | | | | | | | | | Lambda: 3.7406e-06 | | 116 | 4 | Accept | 0.74185 | 2.7544 | 0.2411 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 74.975 | | 117 | 4 | Accept | 0.74185 | 2.6565 | 0.2411 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 74.975 |
| 118 | 4 | Accept | 0.45797 | 11.614 | 0.2411 | 0.24441 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.94716 | | | | | | | | | | KernelScale: 0.072905 |
| 119 | 4 | Accept | 0.65271 | 1.6037 | 0.2411 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010356 | | | | | | | | | | KernelScale: 1.3521 |
| 120 | 4 | Accept | 0.3108 | 1.4953 | 0.2411 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.080708 | | | | | | | | | | KernelScale: 2.7439 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 121 | 4 | Accept | 0.2414 | 1.2954 | 0.2411 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0048396 | | | | | | | | | | KernelScale: 0.02413 |
| 122 | 4 | Accept | 0.5157 | 9.6657 | 0.2411 | 0.24441 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.071659 | | | | | | | | | | Lambda: 1.3447e-05 |
| 123 | 4 | Accept | 0.53186 | 390.32 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 416.46 | | | | | | | | | | KernelScale: 0.0019087 | | 124 | 4 | Best | 0.2402 | 1.4571 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.033468 | | | | | | | | | | KernelScale: 0.073489 |
| 125 | 4 | Accept | 0.2402 | 1.3713 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0040249 | | | | | | | | | | KernelScale: 0.030373 |
| 126 | 3 | Accept | 0.24529 | 39.503 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.32116 | | | | | | | | | | KernelScale: 0.0076281 | | 127 | 3 | Accept | 0.25576 | 1.4031 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.018022 | | | | | | | | | | KernelScale: 0.3703 |
| 128 | 6 | Accept | 0.2426 | 1.3555 | 0.2402 | 0.24229 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.029999 | | | | | | | | | | KernelScale: 0.046245 |
| 129 | 3 | Accept | 0.24499 | 9.5236 | 0.2402 | 0.24419 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.38346 | | | | | | | | | | KernelScale: 0.01786 | | 130 | 3 | Accept | 0.28059 | 0.12828 | 0.2402 | 0.24419 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 131 | 3 | Accept | 0.6886 | 1.4213 | 0.2402 | 0.24419 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.0763 | | | | | | | | | | Learner: logistic | | 132 | 3 | Accept | 0.4487 | 3.111 | 0.2402 | 0.24419 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 4.845 | | | | | | | | | | KernelScale: 0.74028 |
| 133 | 6 | Accept | 0.24469 | 8.3042 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45335 | | | | | | | | | | KernelScale: 0.02034 |
| 134 | 3 | Accept | 0.26204 | 1.7109 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0011217 | | | | | | | | | | KernelScale: 0.10651 | | 135 | 3 | Accept | 0.74185 | 4.0388 | 0.2402 | 0.24441 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 10.518 | | | | | | | | | | Lambda: 0.20458 | | 136 | 3 | Accept | 0.29734 | 9.9598 | 0.2402 | 0.24441 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 107 | | | | | | | | | | LearnRate: 0.0023672 | | | | | | | | | | MinLeafSize: 197 | | 137 | 3 | Accept | 0.44391 | 2.0399 | 0.2402 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.00025608 |
| 138 | 6 | Accept | 0.24559 | 3.2102 | 0.2402 | 0.24276 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.5211 | | | | | | | | | | KernelScale: 0.04163 |
| 139 | 4 | Accept | 0.25067 | 1.4995 | 0.2402 | 0.24276 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.030306 | | | | | | | | | | KernelScale: 0.37391 | | 140 | 4 | Accept | 0.24559 | 1.6227 | 0.2402 | 0.24276 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.0973e-07 | | | | | | | | | | Learner: svm | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 141 | 4 | Accept | 0.74634 | 7.0945 | 0.2402 | 0.24276 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.01094 | | | | | | | | | | Lambda: 0.0013866 |
| 142 | 4 | Accept | 0.29076 | 1.3294 | 0.2402 | 0.24334 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.006875 | | | | | | | | | | KernelScale: 0.38629 |
| 143 | 4 | Accept | 0.24379 | 1.4595 | 0.2402 | 0.24269 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010315 | | | | | | | | | | KernelScale: 0.0079737 |
| 144 | 4 | Accept | 0.24499 | 4.9134 | 0.2402 | 0.24226 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.62692 | | | | | | | | | | KernelScale: 0.033311 |
| 145 | 4 | Accept | 0.24469 | 12.208 | 0.2402 | 0.24242 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010122 | | | | | | | | | | KernelScale: 0.0010167 |
| 146 | 4 | Accept | 0.38588 | 1.5037 | 0.2402 | 0.24388 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0011127 | | | | | | | | | | KernelScale: 0.52838 |
| 147 | 4 | Accept | 0.24589 | 1.3026 | 0.2402 | 0.24228 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.1424 | | | | | | | | | | KernelScale: 0.58035 |
| 148 | 4 | Accept | 0.2408 | 1.2582 | 0.2402 | 0.24165 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010168 | | | | | | | | | | KernelScale: 0.032668 |
| 149 | 4 | Accept | 0.24469 | 1.2764 | 0.2402 | 0.24197 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.13008 | | | | | | | | | | KernelScale: 0.53455 |
| 150 | 5 | Accept | 0.24469 | 6.7111 | 0.2402 | 0.24222 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45516 | | | | | | | | | | KernelScale: 0.024796 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 151 | 5 | Accept | 0.30422 | 1.5471 | 0.2402 | 0.24244 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.36805 | | | | | | | | | | KernelScale: 4.7124 |
| 152 | 6 | Accept | 0.24559 | 1.5701 | 0.2402 | 0.2414 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.21039 | | | | | | | | | | KernelScale: 0.63504 |
| 153 | 5 | Accept | 0.24529 | 66.311 | 0.2402 | 0.24243 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45714 | | | | | | | | | | KernelScale: 0.0069546 | | 154 | 5 | Accept | 0.24589 | 1.604 | 0.2402 | 0.24243 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010712 | | | | | | | | | | KernelScale: 0.043866 |
| 155 | 5 | Accept | 0.29345 | 1.4823 | 0.2402 | 0.24262 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.032053 | | | | | | | | | | KernelScale: 0.88175 |
| 156 | 6 | Accept | 0.25247 | 1.4792 | 0.2402 | 0.24289 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.14103 | | | | | | | | | | KernelScale: 0.91489 |
| 157 | 6 | Accept | 0.2405 | 1.5732 | 0.2402 | 0.242 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010011 | | | | | | | | | | KernelScale: 0.012195 |
| 158 | 6 | Accept | 0.2426 | 1.5232 | 0.2402 | 0.24223 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.52301 | | | | | | | | | | KernelScale: 0.69511 |
| 159 | 6 | Accept | 0.25456 | 1.9467 | 0.2402 | 0.24253 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.63305 | | | | | | | | | | KernelScale: 2.1073 |
| 160 | 5 | Accept | 0.24559 | 133.37 | 0.2402 | 0.24253 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.73914 | | | | | | | | | | KernelScale: 0.005832 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 161 | 5 | Accept | 0.57314 | 8.1723 | 0.2402 | 0.24253 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.071267 | | | | | | | | | | Lambda: 1.4009e-06 |
| 162 | 5 | Accept | 0.24649 | 1.6878 | 0.2402 | 0.24325 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.19825 | | | | | | | | | | KernelScale: 0.74742 |
| 163 | 5 | Accept | 0.2417 | 1.5824 | 0.2402 | 0.24297 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010619 | | | | | | | | | | KernelScale: 0.01008 |
| 164 | 4 | Accept | 0.45528 | 140.46 | 0.2402 | 0.24441 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.00706 | | | | | | | | | | KernelScale: 0.030909 | | 165 | 4 | Accept | 0.31409 | 1.5573 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.56217 | | | | | | | | | | KernelScale: 7.5961 |
| 166 | 4 | Accept | 0.46575 | 2.2317 | 0.2402 | 0.24441 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.10607 | | | | | | | | | | KernelScale: 0.67446 |
| 167 | 4 | Accept | 0.24529 | 3.3215 | 0.2402 | 0.24334 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.4843 | | | | | | | | | | KernelScale: 0.069679 |
| 168 | 4 | Accept | 0.25396 | 1.3297 | 0.2402 | 0.24353 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.52671 | | | | | | | | | | KernelScale: 1.9394 |
| 169 | 5 | Accept | 0.2402 | 1.5257 | 0.2402 | 0.24313 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010378 | | | | | | | | | | KernelScale: 0.012749 |
| 170 | 6 | Accept | 0.3464 | 1.8583 | 0.2402 | 0.24237 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.50591 | | | | | | | | | | KernelScale: 9.2835 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 171 | 6 | Accept | 0.24499 | 12.298 | 0.2402 | 0.24203 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0018442 | | | | | | | | | | KernelScale: 0.0012506 |
| 172 | 6 | Accept | 0.30272 | 1.6031 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.8265 | | | | | | | | | | KernelScale: 11.897 |
| 173 | 6 | Accept | 0.24349 | 1.4247 | 0.2402 | 0.24134 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010459 | | | | | | | | | | KernelScale: 0.036404 |
| 174 | 6 | Accept | 0.29794 | 1.6502 | 0.2402 | 0.24299 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 5.4893 | | | | | | | | | | KernelScale: 13.359 |
| 175 | 5 | Accept | 0.24529 | 109.43 | 0.2402 | 0.24345 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.88969 | | | | | | | | | | KernelScale: 0.0075436 | | 176 | 5 | Accept | 0.25007 | 1.6406 | 0.2402 | 0.24345 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010212 | | | | | | | | | | KernelScale: 0.071763 |
| 177 | 5 | Accept | 0.24619 | 2.171 | 0.2402 | 0.24259 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010341 | | | | | | | | | | KernelScale: 0.0039146 |
| 178 | 5 | Accept | 0.32576 | 1.6841 | 0.2402 | 0.24185 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.6102 | | | | | | | | | | KernelScale: 14.532 |
| 179 | 5 | Accept | 0.8262 | 890.49 | 0.2402 | 0.24179 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.18649 | | | | | | | | | | KernelScale: 0.0010802 |
| 180 | 5 | Accept | 0.30212 | 1.5671 | 0.2402 | 0.24174 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 6.0833 | | | | | | | | | | KernelScale: 15.913 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 181 | 5 | Accept | 0.24499 | 14.697 | 0.2402 | 0.24192 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0078062 | | | | | | | | | | KernelScale: 0.0020664 |
| 182 | 5 | Accept | 0.24529 | 61.944 | 0.2402 | 0.2414 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.0466 | | | | | | | | | | KernelScale: 0.011265 |
| 183 | 5 | Accept | 0.24499 | 17.644 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0021279 | | | | | | | | | | KernelScale: 0.0010743 |
| 184 | 5 | Accept | 0.29794 | 1.6007 | 0.2402 | 0.24189 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010132 | | | | | | | | | | KernelScale: 0.17669 |
| 185 | 4 | Accept | 0.24918 | 279.94 | 0.2402 | 0.24236 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 891.39 | | | | | | | | | | KernelScale: 0.076868 | | 186 | 4 | Accept | 0.26593 | 1.4006 | 0.2402 | 0.24236 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.14381 | | | | | | | | | | KernelScale: 1.2859 |
| 187 | 4 | Accept | 0.24768 | 1.438 | 0.2402 | 0.24161 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.47763 | | | | | | | | | | KernelScale: 1.4081 |
| 188 | 4 | Accept | 0.24589 | 2.5142 | 0.2402 | 0.2416 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010362 | | | | | | | | | | KernelScale: 0.0024579 |
| 189 | 4 | Accept | 0.2405 | 1.2662 | 0.2402 | 0.24215 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 11.232 | | | | | | | | | | KernelScale: 1.4768 |
| 190 | 4 | Accept | 0.24678 | 1.2604 | 0.2402 | 0.24174 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.48152 | | | | | | | | | | KernelScale: 1.1771 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 191 | 4 | Accept | 0.25426 | 1.2846 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.47194 | | | | | | | | | | KernelScale: 1.7714 |
| 192 | 5 | Accept | 0.24768 | 1.3295 | 0.2402 | 0.24188 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.44397 | | | | | | | | | | KernelScale: 1.3493 |
| 193 | 6 | Accept | 0.2414 | 1.3979 | 0.2402 | 0.24154 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.8796 | | | | | | | | | | KernelScale: 1.4211 |
| 194 | 6 | Accept | 0.24499 | 18.984 | 0.2402 | 0.24225 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0018335 | | | | | | | | | | KernelScale: 0.0010104 |
| 195 | 6 | Accept | 0.25037 | 1.5112 | 0.2402 | 0.24212 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.41413 | | | | | | | | | | KernelScale: 1.4211 |
| 196 | 6 | Accept | 0.24499 | 6.007 | 0.2402 | 0.24191 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010001 | | | | | | | | | | KernelScale: 0.0013198 |
| 197 | 6 | Accept | 0.24529 | 111.16 | 0.2402 | 0.24215 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.046722 | | | | | | | | | | KernelScale: 0.0016388 |
| 198 | 5 | Accept | 0.24499 | 18.688 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0033281 | | | | | | | | | | KernelScale: 0.0012057 | | 199 | 5 | Accept | 0.2426 | 1.4789 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.0066 | | | | | | | | | | KernelScale: 1.3889 |
| 200 | 6 | Accept | 0.2417 | 1.499 | 0.2402 | 0.24212 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.7327 | | | | | | | | | | KernelScale: 1.3637 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 201 | 5 | Accept | 0.26473 | 277.53 | 0.2402 | 0.24176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.19781 | | | | | | | | | | KernelScale: 0.0010905 | | 202 | 5 | Accept | 0.30332 | 1.5632 | 0.2402 | 0.24176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.075228 | | | | | | | | | | KernelScale: 1.8677 |
| 203 | 5 | Accept | 0.3108 | 1.5519 | 0.2402 | 0.24188 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.8086 | | | | | | | | | | KernelScale: 16.078 |
| 204 | 5 | Accept | 0.2414 | 1.4413 | 0.2402 | 0.24147 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.1547 | | | | | | | | | | KernelScale: 0.57043 |
| 205 | 5 | Accept | 0.3108 | 1.5576 | 0.2402 | 0.24195 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.0235 | | | | | | | | | | KernelScale: 16.787 |
| 206 | 5 | Accept | 0.2411 | 1.3939 | 0.2402 | 0.24163 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010447 | | | | | | | | | | KernelScale: 0.022855 |
| 207 | 5 | Accept | 0.29704 | 1.5568 | 0.2402 | 0.24155 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.11574 | | | | | | | | | | KernelScale: 1.8573 |
| 208 | 5 | Accept | 0.2426 | 1.4114 | 0.2402 | 0.24195 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.2749 | | | | | | | | | | KernelScale: 1.2395 |
| 209 | 5 | Accept | 0.24559 | 3.445 | 0.2402 | 0.2417 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 560.48 | | | | | | | | | | KernelScale: 1.4181 |
| 210 | 4 | Accept | 0.24678 | 266.64 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 11.849 | | | | | | | | | | KernelScale: 0.0095181 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 211 | 4 | Accept | 0.24589 | 2.1278 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 41.82 | | | | | | | | | | KernelScale: 0.71147 |
| 212 | 3 | Accept | 0.95034 | 1042.9 | 0.2402 | 0.24153 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 10.672 | | | | | | | | | | KernelScale: 0.0013453 | | 213 | 3 | Accept | 0.24649 | 2.0006 | 0.2402 | 0.24153 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 210.22 | | | | | | | | | | KernelScale: 1.4654 |
| 214 | 6 | Accept | 0.26653 | 1.2521 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.16417 | | | | | | | | | | KernelScale: 1.3626 |
| 215 | 4 | Accept | 0.2429 | 1.5969 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010295 | | | | | | | | | | KernelScale: 0.008743 | | 216 | 4 | Accept | 0.33353 | 1.2604 | 0.2402 | 0.24136 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0017992 | | 217 | 4 | Accept | 0.28059 | 0.2661 | 0.2402 | 0.24136 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 218 | 4 | Accept | 0.24798 | 1.3024 | 0.2402 | 0.24173 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 66.132 | | | | | | | | | | KernelScale: 16.829 |
| 219 | 4 | Accept | 0.24529 | 1.255 | 0.2402 | 0.24156 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.0858 | | | | | | | | | | KernelScale: 1.3599 |
| 220 | 4 | Accept | 0.2408 | 1.2539 | 0.2402 | 0.24183 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.8001 | | | | | | | | | | KernelScale: 1.3322 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 221 | 4 | Accept | 0.24589 | 1.8906 | 0.2402 | 0.24185 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 399.07 | | | | | | | | | | KernelScale: 2.1791 |
| 222 | 4 | Accept | 0.2414 | 1.2821 | 0.2402 | 0.24187 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 4.8589 | | | | | | | | | | KernelScale: 1.3542 |
| 223 | 4 | Accept | 0.2423 | 1.2804 | 0.2402 | 0.24131 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.6082 | | | | | | | | | | KernelScale: 1.3565 |
| 224 | 4 | Accept | 0.24559 | 3.1158 | 0.2402 | 0.24116 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 536.1 | | | | | | | | | | KernelScale: 1.4679 |
| 225 | 4 | Accept | 0.25067 | 1.3228 | 0.2402 | 0.24178 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 74.127 | | | | | | | | | | KernelScale: 18.528 |
| 226 | 3 | Accept | 0.28358 | 328 | 0.2402 | 0.24207 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.47384 | | | | | | | | | | KernelScale: 0.001006 | | 227 | 3 | Accept | 0.24499 | 6.2167 | 0.2402 | 0.24207 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0026293 | | | | | | | | | | KernelScale: 0.0019141 |
| 228 | 6 | Accept | 0.2402 | 1.2501 | 0.2402 | 0.24173 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010356 | | | | | | | | | | KernelScale: 0.012867 |
| 229 | 3 | Accept | 0.24589 | 1.4248 | 0.2402 | 0.24173 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 104.21 | | | | | | | | | | KernelScale: 17.45 | | 230 | 3 | Accept | 0.43015 | 0.13765 | 0.2402 | 0.24173 | discr | Delta: 0.00951 | | | | | | | | | | Gamma: 0.68613 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 231 | 3 | Accept | 0.47383 | 2.5354 | 0.2402 | 0.24173 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 1.2431 | | | | | | | | | | KernelScale: 0.76632 | | 232 | 3 | Accept | 0.67664 | 0.30377 | 0.2402 | 0.24173 | knn | NumNeighbors: 8 | | | | | | | | | | Distance: jaccard |
| 233 | 6 | Accept | 0.2414 | 1.2369 | 0.2402 | 0.24133 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 820.98 | | | | | | | | | | KernelScale: 17.331 |
| 234 | 3 | Accept | 0.27759 | 1.4452 | 0.2402 | 0.24133 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 22.198 | | | | | | | | | | KernelScale: 17.97 | | 235 | 3 | Accept | 0.28059 | 0.13623 | 0.2402 | 0.24133 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | 236 | 3 | Accept | 0.52408 | 1.503 | 0.2402 | 0.24133 | knn | NumNeighbors: 351 | | | | | | | | | | Distance: mahalanobis | | 237 | 3 | Accept | 0.27789 | 5.8867 | 0.2402 | 0.24133 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 68 | | | | | | | | | | LearnRate: 0.18331 | | | | | | | | | | MinLeafSize: 3 |
| 238 | 6 | Accept | 0.24499 | 1.2819 | 0.2402 | 0.24188 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 201.86 | | | | | | | | | | KernelScale: 18.062 |
| 239 | 3 | Accept | 0.25097 | 1.4345 | 0.2402 | 0.24116 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 105.8 | | | | | | | | | | KernelScale: 23.315 | | 240 | 3 | Accept | 0.30212 | 0.12275 | 0.2402 | 0.24116 | tree | MinLeafSize: 122 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 241 | 3 | Accept | 0.25307 | 11.978 | 0.2402 | 0.24116 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 119 | | | | | | | | | | LearnRate: 0.60308 | | | | | | | | | | MinLeafSize: 1 | | 242 | 3 | Accept | 0.44182 | 11.435 | 0.2402 | 0.24116 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.2783 | | | | | | | | | | KernelScale: 0.19364 |
__________________________________________________________ Optimization completed. Total iterations: 242 Total elapsed time: 1907.7403 seconds Total time for training and validation: 4936.5339 seconds Best observed learner is a multiclass svm model with: Coding (ECOC): onevsone BoxConstraint: 0.033468 KernelScale: 0.073489 Observed validation loss: 0.2402 Time for training and validation: 1.4571 seconds Best estimated learner (returned model) is a multiclass svm model with: Coding (ECOC): onevsone BoxConstraint: 0.0010378 KernelScale: 0.012749 Estimated validation loss: 0.24116 Estimated time for training and validation: 1.5595 seconds Documentation for fitcauto display
The final model returned by fitcauto
corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (creditTrain
), the listed Learner
(or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
The model Mdl
corresponds to the best point in the Bayesian optimization according to the 'min-visited-mean'
criterion. To gauge how the model will perform on new data, look at the observed cross-validation accuracy of the model (cvAccuracy
) and its general estimated performance based on the Bayesian optimization (estimatedAccuracy
).
[x,~,iteration] = bestPoint(Results,'Criterion','min-visited-mean'); cvError = Results.ObjectiveTrace(iteration); cvAccuracy = 1 - cvError
cvAccuracy = 0.7598
estimatedError = predictObjective(Results,x); estimatedAccuracy = 1 - estimatedError
estimatedAccuracy = 0.7588
Evaluate the performance of the model on the test set. Create a confusion matrix from the results, and specify the order of the classes in the confusion matrix.
testAccuracy = 1 - loss(Mdl,creditTest,'Rating')
testAccuracy = 0.7438
cm = confusionchart(creditTest.Rating,predict(Mdl,creditTest)); sortClasses(cm,{'AAA','AA','A','BBB','BB','B','CCC'})
Tbl
— Sample dataSample data, specified as a table. Each row of Tbl
corresponds to one observation, and each column corresponds to one predictor. Optionally, Tbl
can contain one additional column for the response variable. Multicolumn variables and cell arrays other than cell arrays of character vectors are not accepted.
If Tbl
contains the response variable, and you want to use all remaining
variables in Tbl
as predictors, specify the response variable using
ResponseVarName
.
If Tbl
contains the response variable, and you want to use only a subset of the remaining variables in Tbl
as predictors, specify a formula using formula
.
If Tbl
does not contain the response variable, specify a response variable using Y
. The length of the response variable and the number of rows in Tbl
must be equal.
Data Types: table
ResponseVarName
— Response variable nameTbl
Response variable name, specified as the name of a variable in
Tbl
.
You must specify ResponseVarName
as a character vector or string scalar.
For example, if the response variable Y
is
stored as Tbl.Y
, then specify it as
'Y'
. Otherwise, the software
treats all columns of Tbl
, including
Y
, as predictors when training
the model.
The response variable must be a categorical, character, or string array; a logical or numeric
vector; or a cell array of character vectors. If
Y
is a character array, then each
element of the response variable must correspond to one row of
the array.
A good practice is to specify the order of the classes by using the
ClassNames
name-value
argument.
Data Types: char
| string
formula
— Explanatory model of response variable and subset of predictor variablesExplanatory model of the response variable and a subset of the predictor variables,
specified as a character vector or string scalar in the form
'Y~x1+x2+x3'
. In this form, Y
represents the
response variable, and x1
, x2
, and
x3
represent the predictor variables.
To specify a subset of variables in Tbl
as predictors for
training the model, use a formula. If you specify a formula, then the software does not
use any variables in Tbl
that do not appear in
formula
.
The variable names in the formula must be both variable names in Tbl
(Tbl.Properties.VariableNames
) and valid MATLAB® identifiers. You can verify the variable names in Tbl
by
using the isvarname
function. If the variable names
are not valid, then you can convert them by using the matlab.lang.makeValidName
function.
Data Types: char
| string
Y
— Class labelsClass labels, specified as a numeric, categorical, or logical vector, a character or string array, or a cell array of character vectors.
If Y
is a character array, then each element of the class
labels must correspond to one row of the array.
The length of Y
must be equal to the number of rows in
Tbl
or X
.
A good practice is to specify the class order by using the
ClassNames
name-value pair argument.
Data Types: single
| double
| categorical
| logical
| char
| string
| cell
X
— Predictor dataPredictor data, specified as a numeric matrix.
Each row of X
corresponds to one observation, and each column corresponds to one predictor.
The length of Y
and the number of rows in X
must be equal.
To specify the names of the predictors in the order of their appearance in X
, use the PredictorNames
name-value pair argument.
Data Types: single
| double
Note
The software treats NaN
, empty character vector
(''
), empty string (""
),
<missing>
, and <undefined>
elements as
missing data. The software removes rows of data corresponding to missing values in the
response variable. However, the treatment of missing values in the predictor data
X
or Tbl
varies among models (or
learners).
Specify optional
comma-separated pairs of Name,Value
arguments. Name
is
the argument name and Value
is the corresponding value.
Name
must appear inside quotes. You can specify several name and value
pair arguments in any order as
Name1,Value1,...,NameN,ValueN
.
'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations',200,'Verbose',2)
specifies to run 200 iterations of the optimization process (that is, try 200 model
hyperparameter combinations), and to display information in the Command Window about the
next model hyperparameter combination to be evaluated.'Learners'
— Types of classification models'auto'
(default) | 'all'
| 'all-linear'
| 'all-nonlinear'
| one or more learner namesTypes of classification models to try during the optimization, specified as the
comma-separated pair consisting of 'Learners'
and a value in the
first table below or one or more learner names in the second table. Specify multiple
learner names as a string or cell array.
Value | Description |
---|---|
'auto' | fitcauto automatically selects a subset of
learners, suitable for the given predictor and response data. The learners
can have model hyperparameter values that differ from the default. For more
information, see Automatic Selection of Learners. |
'all' | fitcauto selects all possible learners. |
'all-linear' | fitcauto selects linear learners:
'discr' (with a linear discriminant type) and
'linear' . |
'all-nonlinear' | fitcauto selects all nonlinear learners:
'discr' (with a quadratic discriminant type),
'ensemble' , 'kernel' ,
'knn' , 'nb' ,
'svm' (with a Gaussian or polynomial kernel), and
'tree' . |
Note
For greater efficiency, fitcauto
does not select the following combinations of models when you specify one of the previous values.
'kernel'
and 'svm'
(with a Gaussian
kernel) — fitcauto
chooses the first when the predictor
data has more than 11,000 observations, and the second otherwise.
'linear'
and 'svm'
(with a linear
kernel) — fitcauto
chooses the first.
Learner Name | Description |
---|---|
'discr' | Discriminant analysis classifier |
'ensemble' | Ensemble classification model |
'kernel' | Kernel classification model |
'knn' | k-nearest neighbor model |
'linear' | Linear classification model |
'nb' | Naive Bayes classifier |
'svm' | Support vector machine classifier |
'tree' | Binary decision classification tree |
Example: 'Learners','all'
Example: 'Learners','ensemble'
Example: 'Learners',{'svm','tree'}
Data Types: char
| string
| cell
'OptimizeHyperparameters'
— Hyperparameters to optimize'auto'
(default) | 'all'
Hyperparameters to optimize, specified as the comma-separated pair consisting of
'OptimizeHyperparameters'
and 'auto'
or
'all'
. The optimizable hyperparameters depend on the model (or
learner), as described in this table.
Learner Name | Hyperparameters for 'auto' | Additional Hyperparameters for 'all' | Notes |
---|---|---|---|
'discr' | Delta , Gamma | DiscrimType |
For more information, including hyperparameter search
ranges, see |
'ensemble' | Method , NumLearningCycles , LearnRate , MinLeafSize | MaxNumSplits ,
NumVariablesToSample , SplitCriterion | When the ensemble For more information, including
hyperparameter search ranges, see |
'kernel' | KernelScale , Lambda , Coding (for three
or more classes only) | Learner , NumExpansionDimensions | For more information, including hyperparameter search ranges, see
OptimizeHyperparameters and OptimizeHyperparameters (for three or more classes only). Note
that you cannot change hyperparameter search ranges when you use
fitcauto . |
'knn' | Distance ,
NumNeighbors | DistanceWeight , Exponent ,
Standardize | For more information, including hyperparameter search ranges, see
OptimizeHyperparameters . Note that you cannot change
hyperparameter search ranges when you use
fitcauto . |
'linear' | Lambda , Learner , Coding (for three
or more classes only) | Regularization | For more information, including hyperparameter search ranges, see
OptimizeHyperparameters and OptimizeHyperparameters (for three or more classes only). Note
that you cannot change hyperparameter search ranges when you use
fitcauto . |
'nb' | DistributionNames , Width | Kernel | For more information, including hyperparameter search ranges, see
OptimizeHyperparameters . Note that you cannot change
hyperparameter search ranges when you use
fitcauto . |
'svm' | BoxConstraint , KernelScale ,
Coding (for three
or more classes only) | KernelFunction , PolynomialOrder , Standardize |
For more information, including hyperparameter search
ranges, see |
'tree' | MinLeafSize | MaxNumSplits ,
SplitCriterion | For more information, including hyperparameter search ranges, see
OptimizeHyperparameters . Note that you cannot change
hyperparameter search ranges when you use
fitcauto . |
Note
When 'Learners'
is set to a value other than
'auto'
, the default values for the model hyperparameters not
being optimized match the default fit function values, unless otherwise indicated in
the table notes. When 'Learners'
is set to
'auto'
, the optimized hyperparameter search ranges and
nonoptimized hyperparameter values can vary, depending on the characteristics of the
training data. For more information, see Automatic Selection of Learners.
Example: 'OptimizeHyperparameters','all'
'HyperparameterOptimizationOptions'
— Options for optimizationOptions for the optimization, specified as the comma-separated pair consisting of
'HyperparameterOptimizationOptions'
and a structure. All fields
in the structure are optional.
Field Name | Values | Default |
---|---|---|
MaxObjectiveEvaluations | Maximum number of iterations (objective function evaluations) | 30*L , where L is the number of
learners (see Learners ) |
MaxTime | Time limit, specified as a positive real number. The time limit is
in seconds, as measured by | Inf |
ShowPlots | Logical value indicating whether to show plots. If
true , this field plots the best observed and estimated
objective function values (so far) against the iteration number. | true |
SaveIntermediateResults | Logical value indicating whether to save results. If
true , this field overwrites a workspace variable named
'BayesoptResults' at each iteration. The variable is a
BayesianOptimization object. | false |
Verbose | Display at the command line:
| 1 |
UseParallel | Logical value indicating whether to run Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. | false |
Repartition | Logical value indicating whether to repartition the
cross-validation at every iteration. If
| false |
Specify only one of the following three options. | ||
CVPartition | cvpartition object, created by cvpartition | 'Kfold',5 if you do not specify any
cross-validation field |
Holdout | Scalar in the range (0,1) representing the holdout
fraction | |
Kfold | Integer greater than 1 |
Example: 'HyperparameterOptimizationOptions',struct('UseParallel',true)
Data Types: struct
'CategoricalPredictors'
— Categorical predictors list'all'
Categorical predictors list, specified as one of the values in this table.
Value | Description |
---|---|
Vector of positive integers |
Each entry in the vector is an index value corresponding to the column of the predictor data that contains a categorical variable. The index values are between 1 and If |
Logical vector |
A |
Character matrix | Each row of the matrix is the name of a predictor variable. The names must match the entries in PredictorNames . Pad the names with extra blanks so each row of the character matrix has the same length. |
String array or cell array of character vectors | Each element in the array is the name of a predictor variable. The names must match the entries in PredictorNames . |
'all' | All predictors are categorical. |
By default, if the predictor data is in a table (Tbl
),
fitcauto
assumes that a variable is categorical if it is a
logical vector, categorical vector, character array, string array, or cell array of
character vectors. However, learners that use decision trees assume that mathematically
ordered categorical vectors are continuous variables. If the predictor data is a matrix
(X
), fitcauto
assumes that all
predictors are continuous. To identify any other predictors as categorical predictors,
specify them by using the 'CategoricalPredictors'
name-value pair
argument.
For more information on how fitting functions treat categorical predictors, see Automatic Creation of Dummy Variables.
Note
fitcauto
does not support categorical predictors
for discriminant analysis classifiers. That is, if you want
Learners
to include 'discr'
models, you cannot specify the 'CategoricalPredictors'
name-value pair argument or use a table of sample data
(Tbl
) containing categorical predictors.
fitcauto
does not support a mix of numeric and
categorical predictors for k-nearest neighbor models. That is, if you want
Learners
to include 'knn'
models, you must specify the 'CategoricalPredictors'
value as 'all'
or []
.
Example: 'CategoricalPredictors','all'
Data Types: single
| double
| logical
| char
| string
| cell
'ClassNames'
— Names of classes to use for trainingNames of classes to use for training, specified as a categorical, character, or string array;
a logical or numeric vector; or a cell array of character vectors.
ClassNames
must have the same data type as the response variable
in Tbl
or Y
.
If ClassNames
is a character array, then each element must correspond to
one row of the array.
Use ClassNames
to:
Specify the order of the classes during training.
Specify the order of any input or output argument dimension that
corresponds to the class order. For example, use
ClassNames
to specify the order of the dimensions of
Cost
or the column order of classification scores
returned by predict
.
Select a subset of classes for training. For example, suppose that the set
of all distinct class names in Y
is
{'a','b','c'}
. To train the model using observations
from classes 'a'
and 'c'
only, specify
'ClassNames',{'a','c'}
.
The default value for ClassNames
is the set of all distinct class names in
the response variable in Tbl
or Y
.
Example: 'ClassNames',{'b','g'}
Data Types: categorical
| char
| string
| logical
| single
| double
| cell
'Cost'
— Misclassification costMisclassification cost, specified as the comma-separated pair consisting of
'Cost'
and a square matrix or structure array.
If you specify a square matrix Cost
and the true class of
an observation is i
, then Cost(i,j)
is the
cost of classifying a point into class j
. That is, rows
correspond to the true classes and columns correspond to the predicted classes. To
specify the class order for the corresponding rows and columns of
Cost
, also specify the ClassNames
name-value pair argument.
If you specify a structure S
, then it must have two fields:
S.ClassNames
, which contains the class names as a
variable of the same data type as Y
S.ClassificationCosts
, which contains the cost matrix
with rows and columns ordered as in S.ClassNames
The default value for Cost
is ones(K) –
eye(K)
, where K
is the number of distinct
classes.
Example: 'Cost',[0 1; 2 0]
Data Types: single
| double
| struct
'PredictorNames'
— Predictor variable namesPredictor variable names, specified as a string array of unique names or cell array of unique
character vectors. The functionality of PredictorNames
depends on the
way you supply the training data.
If you supply X
and Y
, then you
can use PredictorNames
to assign names to the predictor
variables in X
.
The order of the names in PredictorNames
must correspond to the column order of X
.
That is, PredictorNames{1}
is the name of
X(:,1)
,
PredictorNames{2}
is the name of
X(:,2)
, and so on. Also,
size(X,2)
and
numel(PredictorNames)
must be
equal.
By default, PredictorNames
is
{'x1','x2',...}
.
If you supply Tbl
, then you can use
PredictorNames
to choose which predictor variables to
use in training. That is, fitcauto
uses only the
predictor variables in PredictorNames
and the response
variable during training.
PredictorNames
must be a subset of
Tbl.Properties.VariableNames
and cannot
include the name of the response variable.
By default, PredictorNames
contains the
names of all predictor variables.
A good practice is to specify the predictors for training
using either 'PredictorNames'
or
formula
, but not both.
Example: 'PredictorNames',{'SepalLength','SepalWidth','PetalLength','PetalWidth'}
Data Types: string
| cell
'Prior'
— Prior probabilities'empirical'
(default) | 'uniform'
| numeric vector | structure arrayPrior probabilities for each class, specified as the comma-separated pair
consisting of 'Prior'
and a value in this table.
Value | Description |
---|---|
'empirical' | The class prior probabilities are the class relative frequencies in
Y . |
'uniform' | All class prior probabilities are equal to 1/K, where K is the number of classes. |
numeric vector | Each element is a class prior probability. Order the elements according
to Mdl .ClassNames or specify the
order using the ClassNames name-value pair argument.
The software normalizes the elements to sum to 1 . |
structure | A structure
|
Example: 'Prior',struct('ClassNames',{{'b','g'}},'ClassProbs',1:2)
Data Types: single
| double
| char
| string
| struct
'ResponseName'
— Response variable name'Y'
(default) | character vector | string scalarResponse variable name, specified as a character vector or string scalar.
If you supply Y
, then you can
use 'ResponseName'
to specify a name for the response
variable.
If you supply ResponseVarName
or formula
,
then you cannot use 'ResponseName'
.
Example: 'ResponseName','response'
Data Types: char
| string
'ScoreTransform'
— Score transformation'none'
(default) | 'doublelogit'
| 'invlogit'
| 'ismax'
| 'logit'
| function handle | ...Score transformation, specified as a character vector, string scalar, or function handle.
This table summarizes the available character vectors and string scalars.
Value | Description |
---|---|
'doublelogit' | 1/(1 + e–2x) |
'invlogit' | log(x / (1 – x)) |
'ismax' | Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0 |
'logit' | 1/(1 + e–x) |
'none' or 'identity' | x (no transformation) |
'sign' | –1 for x < 0 0 for x = 0 1 for x > 0 |
'symmetric' | 2x – 1 |
'symmetricismax' | Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1 |
'symmetriclogit' | 2/(1 + e–x) – 1 |
For a MATLAB function or a function you define, use its function handle for the score transform. The function handle must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).
Example: 'ScoreTransform','logit'
Data Types: char
| string
| function_handle
'Weights'
— Observation weightsTbl
Observation weights, specified as the comma-separated pair consisting of
'Weights'
and a positive numeric vector or the name of a variable
in Tbl
. The software weights each observation in
X
or Tbl
with the corresponding value in
Weights
. The length of Weights
must equal
the number of rows in X
or Tbl
.
If you specify the input data as a table Tbl
, then
Weights
can be the name of a variable in
Tbl
that contains a numeric vector. In this case, you must
specify Weights
as a character vector or string scalar. For
example, if the weights vector W
is stored as
Tbl.W
, then specify it as 'W'
. Otherwise, the
software treats all columns of Tbl
, including
W
, as predictors or the response variable when training the
model.
By default, Weights
is ones(n,1)
, where
n
is the number of observations in X
or
Tbl
.
The software normalizes Weights
to sum to the value of the
prior probability in the respective class.
Data Types: single
| double
| char
| string
Mdl
— Trained classification modelTrained classification model, returned as one of the classification model objects in this table.
Learner Name | Returned Model Object |
---|---|
'discr' | CompactClassificationDiscriminant |
'ensemble' | CompactClassificationEnsemble |
'kernel' |
|
'knn' | ClassificationKNN |
'linear' |
|
'nb' | CompactClassificationNaiveBayes |
'svm' |
|
'tree' | CompactClassificationTree |
OptimizationResults
— Optimization resultsBayesianOptimization
objectOptimization results, returned as a BayesianOptimization
object. For more information on the Bayesian
optimization process, see Bayesian Optimization.
When you set the Verbose
field of the
HyperparameterOptimizationOptions
name-value pair argument to
1
or 2
, the fitcauto
function
provides an iterative display of the optimization results.
The following table describes the columns in the display and their entries.
Column Name | Description |
---|---|
Iter | Iteration number — You can set a limit to the number of iterations by using
the MaxObjectiveEvaluations field of the
'HyperparameterOptimizationOptions' name-value pair
argument. |
Active workers | Number of active parallel workers — This column appears only when you run the
optimization in parallel by setting the UseParallel field of
the 'HyperparameterOptimizationOptions' name-value pair
argument to true . |
Eval result | One of the following evaluation results:
|
Validation loss | Validation loss computed for the learner and hyperparameter values at this
iteration — In particular, fitcauto computes the
cross-validation classification error by default. You can change the validation
scheme by using the CVPartition , Holdout , or
Kfold field of the
'HyperparameterOptimizationOptions' name-value pair
argument. |
Time for training & validation (sec) | Time taken to train and compute the validation loss for the model with the learner and hyperparameter values at this iteration (in seconds) — In particular, this value excludes the time required to update the objective function model maintained by the Bayesian optimization process. For more details, see Bayesian Optimization. |
Observed min validation loss | Observed minimum validation loss computed so far — This value
corresponds to the smallest By default,
|
Estimated min validation loss | Estimated minimum validation loss — At each iteration,
By default, |
Learner | Model type evaluated at this iteration — Specify the learners used in the
optimization by using the 'Learners' name-value pair
argument. |
Hyperparameter: Value | Hyperparameter values at this iteration — Specify the hyperparameters used in
the optimization by using the 'OptimizeHyperparameters'
name-value pair argument. |
The display also includes a description of two models:
Best observed learner
— This model, with the listed learner
type and hyperparameter values, yields the final observed minimum validation
loss.
Best estimated learner
— This model, with the listed learner
type and hyperparameter values, yields the final estimated minimum validation loss.
fitcauto
retrains the model on the entire training data set and
returns it as the Mdl
output.
Depending on the size of your data and the number of learners you specify,
fitcauto
can take some time to run. If you have a Parallel Computing Toolbox license, you can speed up computations by running the optimization in
parallel. To do so, specify
'HyperparameterOptimizationOptions',struct('UseParallel',true)
. You
can include other fields in the structure to control other aspects of the optimization.
See HyperparameterOptimizationOptions
.
When you specify 'Learners','auto'
, the fitcauto
function analyzes the predictor and response data in order to choose appropriate learners.
The function considers whether the data set has any of these characteristics:
Categorical predictors
Missing values for more than 5% of the data
Imbalanced data, where the ratio of the number of observations in the largest class to the number of observations in the smallest class is greater than 5
More than 100 observations in the smallest class
Wide data, where the number of predictors is greater than or equal to the number of observations
High-dimensional data, where the number of predictors is greater than 100
Large data, where the number of observations is greater than 50,000
Binary response variable
Ordinal response variable
The selected learners are always a subset of those listed in the
Learners
table. However, the associated models tried during the
optimization process can have different default values for hyperparameters not being
optimized, as well as different search ranges for hyperparameters being optimized.
The goal of Bayesian optimization, and optimization in general, is to find a point that
minimizes an objective function. In the context of fitcauto
, a point is
a learner type together with a set of hyperparameter values for the learner (see Learners
and
OptimizeHyperparameters
), and the objective function is the cross-validation
classification error, by default. The Bayesian optimization implemented in
fitcauto
internally maintains a multi-TreeBagger
model of the objective function. That is, the objective function
model splits along the learner type and, for a given learner, the model is a
TreeBagger
ensemble for regression. (This underlying model differs from
the Gaussian process model employed by other Statistics and Machine Learning Toolbox™ functions that use Bayesian optimization.) Bayesian optimization trains the
underlying model by using objective function evaluations, and determines the next point to
evaluate by using an acquisition function ('expected-improvement'
). For
more information, see Expected Improvement. The acquisition function balances between sampling at
points with low modeled objective function values and exploring areas that are not well
modeled yet. At the end of the optimization, fitcauto
chooses the point
with the minimum objective function model value, among the points evaluated during the
optimization. For more information, see the
'Criterion','min-visited-mean'
name-value pair argument of bestPoint
.
If you are unsure which models work best for your data set, you can alternatively use
the Classification Learner app. Using the app, you can perform hyperparameter
tuning for different models, and choose the optimized model that performs best. Although
you must select a specific model before you can tune the model hyperparameters,
Classification Learner provides greater flexibility for selecting optimizable
hyperparameters and setting hyperparameter values. However, you cannot optimize in
parallel, choose 'linear'
or 'kernel'
learners,
specify observation weights, or specify prior probabilities in the app. For more
information, see Hyperparameter Optimization in Classification Learner App.
If you know which models might suit your data, you can alternatively use the
corresponding model fit functions and specify the
'OptimizeHyperparameters'
name-value pair argument to tune
hyperparameters. You can compare the results across the models to select the best
classifier. For an example of this process, see Moving Towards Automating Model Selection Using Bayesian Optimization.
To perform parallel hyperparameter optimization, use the 'HyperparameterOptimizationOptions',struct('UseParallel',true)
name-value pair argument in the call to this function.
For more general information about parallel computing, see Run MATLAB Functions with Automatic Parallel Support (Parallel Computing Toolbox).
fitcdiscr
| fitcecoc
| fitcensemble
| fitckernel
| fitcknn
| fitclinear
| fitcnb
| fitcsvm
| fitctree
You have a modified version of this example. Do you want to open this example with your edits?
You clicked a link that corresponds to this MATLAB command:
Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
Select web siteYou can also select a web site from the following list:
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.