Main Content
fitcauto
Automatically select classification model with optimized hyperparameters
Syntax
Description
Given predictor and response data, fitcauto automatically
tries a selection of classification model types with different hyperparameter values. The
function uses Bayesian optimization to select models and their hyperparameter values, and
computes the cross-validation classification error for each model. After the optimization is
complete, fitcauto returns the model, trained on the entire data set,
that is expected to best classify new data. You can use the predict and
loss object functions of the returned model to classify new data and
compute the test set classification error, respectively.
Use fitcauto when you are uncertain which classifier types best suit
your data. For information on alternative methods for tuning hyperparameters of classification
models, see Alternative Functionality.
Mdl = fitcauto(Tbl,ResponseVarName)Mdl with tuned hyperparameters. The
table Tbl contains the predictor variables and the response variable,
where ResponseVarName is the name of the response variable.
Mdl = fitcauto(___,Name,Value)HyperparameterOptimizationOptions name-value pair argument to
specify how the Bayesian optimization is performed.
[
additionally returns Mdl,OptimizationResults] = fitcauto(___)OptimizationResults, a
BayesianOptimization object containing the results of the model
selection and hyperparameter tuning process.
Examples
Automatically Select Classifier Using Table Data
Use fitcauto to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in a table.
Load Data
Load the carbig data set, which contains measurements of cars made in the 1970s and early 1980s.
load carbigCategorize the cars based on whether they were made in the USA.
Origin = categorical(cellstr(Origin));
Origin = mergecats(Origin,{'France','Japan','Germany', ...
'Sweden','Italy','England'},'NotUSA');Create a table containing the predictor variables Acceleration, Displacement, and so on, as well as the response variable Origin.
cars = table(Acceleration,Displacement,Horsepower, ...
Model_Year,MPG,Weight,Origin);Partition Data
Partition the data into training and test sets. Use approximately 80% of the observations for the model selection and hyperparameter tuning process, and 20% of the observations to test the performance of the final model returned by fitcauto. Use cvpartition to partition the data.
rng('default') % For reproducibility of the data partition c = cvpartition(Origin,'Holdout',0.2); trainingIdx = training(c); % Training set indices carsTrain = cars(trainingIdx,:); testIdx = test(c); % Test set indices carsTest = cars(testIdx,:);
Run fitcauto
Pass the training data to fitcauto. By default, fitcauto determines appropriate model types to try, uses Bayesian optimization to find good hyperparameter values, and returns a trained model Mdl with the best expected performance. Additionally, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
Expect this process to take some time. To speed up the optimization process, consider specifying to run the optimization in parallel, if you have a Parallel Computing Toolbox™ license. To do so, pass 'HyperparameterOptimizationOptions',struct('UseParallel',true) to fitcauto as a name-value pair argument.
Mdl = fitcauto(carsTrain,'Origin');Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Learner types to explore: ensemble, knn, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 150 Total time (MaxTime): Inf |=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 1 | Best | 0.14154 | 10.563 | 0.14154 | 0.14154 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 201 | | | | | | | | | MinLeafSize: 7 |
| 2 | Accept | 0.18269 | 0.57392 | 0.14154 | 0.14154 | knn | NumNeighbors: 3 |
| 3 | Accept | 0.23397 | 0.1264 | 0.14154 | 0.14154 | knn | NumNeighbors: 91 |
| 4 | Accept | 0.16308 | 12.867 | 0.14154 | 0.15468 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 274 | | | | | | | | | MinLeafSize: 15 |
| 5 | Accept | 0.20833 | 0.124 | 0.14154 | 0.15468 | knn | NumNeighbors: 4 |
| 6 | Accept | 0.22115 | 0.079641 | 0.14154 | 0.15468 | knn | NumNeighbors: 28 |
| 7 | Accept | 0.16923 | 0.20013 | 0.14154 | 0.15468 | tree | MinLeafSize: 105 |
| 8 | Accept | 0.37179 | 0.59222 | 0.14154 | 0.15468 | svm | BoxConstraint: 0.022186 | | | | | | | | | KernelScale: 0.085527 |
| 9 | Accept | 0.37179 | 0.11659 | 0.14154 | 0.15468 | svm | BoxConstraint: 0.045899 | | | | | | | | | KernelScale: 0.0024758 |
| 10 | Accept | 0.24615 | 0.98386 | 0.14154 | 0.15468 | nb | DistributionNames: kernel | | | | | | | | | Width: 1.1327 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 11 | Accept | 0.16923 | 0.079996 | 0.14154 | 0.15468 | tree | MinLeafSize: 78 |
| 12 | Accept | 0.26923 | 0.10923 | 0.14154 | 0.15468 | svm | BoxConstraint: 11.063 | | | | | | | | | KernelScale: 15.114 |
| 13 | Best | 0.12923 | 0.11568 | 0.12923 | 0.15468 | tree | MinLeafSize: 3 |
| 14 | Accept | 0.21154 | 0.084406 | 0.12923 | 0.15468 | knn | NumNeighbors: 2 |
| 15 | Accept | 0.14154 | 0.080022 | 0.12923 | 0.15294 | tree | MinLeafSize: 1 |
| 16 | Accept | 0.14769 | 0.092395 | 0.12923 | 0.15097 | tree | MinLeafSize: 2 |
| 17 | Accept | 0.14154 | 10.869 | 0.12923 | 0.14872 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 208 | | | | | | | | | MinLeafSize: 10 |
| 18 | Accept | 0.37179 | 0.12386 | 0.12923 | 0.14872 | svm | BoxConstraint: 116.46 | | | | | | | | | KernelScale: 0.52908 |
| 19 | Accept | 0.22769 | 0.15545 | 0.12923 | 0.14872 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 20 | Accept | 0.22115 | 0.070813 | 0.12923 | 0.14872 | knn | NumNeighbors: 8 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 21 | Accept | 0.37179 | 0.11553 | 0.12923 | 0.14872 | svm | BoxConstraint: 45.341 | | | | | | | | | KernelScale: 0.76949 |
| 22 | Accept | 0.12923 | 0.080362 | 0.12923 | 0.14194 | tree | MinLeafSize: 3 |
| 23 | Best | 0.10154 | 0.079656 | 0.10154 | 0.13213 | tree | MinLeafSize: 5 |
| 24 | Accept | 0.22769 | 0.2529 | 0.10154 | 0.13213 | nb | DistributionNames: kernel | | | | | | | | | Width: 0.42571 |
| 25 | Accept | 0.11385 | 0.080085 | 0.10154 | 0.1289 | tree | MinLeafSize: 11 |
| 26 | Accept | 0.13782 | 0.092228 | 0.10154 | 0.1289 | svm | BoxConstraint: 9.7286 | | | | | | | | | KernelScale: 293.41 |
| 27 | Accept | 0.22769 | 0.073346 | 0.10154 | 0.1289 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 28 | Accept | 0.21795 | 0.074914 | 0.10154 | 0.1289 | knn | NumNeighbors: 42 |
| 29 | Accept | 0.24308 | 0.27621 | 0.10154 | 0.1289 | nb | DistributionNames: kernel | | | | | | | | | Width: 4.4662 |
| 30 | Accept | 0.16308 | 12.328 | 0.10154 | 0.1289 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 267 | | | | | | | | | MinLeafSize: 131 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 31 | Accept | 0.24308 | 0.22334 | 0.10154 | 0.1289 | nb | DistributionNames: kernel | | | | | | | | | Width: 0.66296 |
| 32 | Accept | 0.22115 | 0.066711 | 0.10154 | 0.1289 | knn | NumNeighbors: 28 |
| 33 | Accept | 0.13846 | 0.079934 | 0.10154 | 0.12465 | tree | MinLeafSize: 25 |
| 34 | Accept | 0.21474 | 0.085438 | 0.10154 | 0.12465 | knn | NumNeighbors: 14 |
| 35 | Accept | 0.16615 | 10.05 | 0.10154 | 0.12465 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 215 | | | | | | | | | MinLeafSize: 13 |
| 36 | Accept | 0.14154 | 12.866 | 0.10154 | 0.12465 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 254 | | | | | | | | | MinLeafSize: 31 |
| 37 | Accept | 0.22769 | 0.070251 | 0.10154 | 0.12465 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 38 | Accept | 0.37179 | 0.077924 | 0.10154 | 0.12465 | svm | BoxConstraint: 0.0073633 | | | | | | | | | KernelScale: 774.33 |
| 39 | Accept | 0.16923 | 0.068411 | 0.10154 | 0.12552 | tree | MinLeafSize: 82 |
| 40 | Accept | 0.20833 | 0.064563 | 0.10154 | 0.12552 | knn | NumNeighbors: 4 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 41 | Accept | 0.16308 | 12.932 | 0.10154 | 0.12552 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 274 | | | | | | | | | MinLeafSize: 150 |
| 42 | Accept | 0.22462 | 0.24365 | 0.10154 | 0.12552 | nb | DistributionNames: kernel | | | | | | | | | Width: 121.64 |
| 43 | Accept | 0.20308 | 11.027 | 0.10154 | 0.12552 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 229 | | | | | | | | | MinLeafSize: 117 |
| 44 | Accept | 0.16923 | 0.069279 | 0.10154 | 0.12291 | tree | MinLeafSize: 84 |
| 45 | Accept | 0.22769 | 0.078716 | 0.10154 | 0.12291 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 46 | Accept | 0.22769 | 0.068458 | 0.10154 | 0.12291 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 47 | Accept | 0.16615 | 9.9849 | 0.10154 | 0.12291 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 212 | | | | | | | | | MinLeafSize: 49 |
| 48 | Accept | 0.14769 | 14.541 | 0.10154 | 0.12291 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 288 | | | | | | | | | MinLeafSize: 25 |
| 49 | Accept | 0.23077 | 0.21379 | 0.10154 | 0.12291 | nb | DistributionNames: kernel | | | | | | | | | Width: 73.249 |
| 50 | Accept | 0.37179 | 0.091937 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.0036501 | | | | | | | | | KernelScale: 1.0504 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 51 | Accept | 0.21474 | 0.098808 | 0.10154 | 0.12291 | svm | BoxConstraint: 64.859 | | | | | | | | | KernelScale: 23.779 |
| 52 | Accept | 0.37179 | 0.10415 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.16622 | | | | | | | | | KernelScale: 4.4901 |
| 53 | Accept | 0.25846 | 0.2444 | 0.10154 | 0.12291 | nb | DistributionNames: kernel | | | | | | | | | Width: 0.079498 |
| 54 | Accept | 0.21154 | 0.074835 | 0.10154 | 0.12291 | knn | NumNeighbors: 2 |
| 55 | Accept | 0.13846 | 12.173 | 0.10154 | 0.12291 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | MinLeafSize: 8 |
| 56 | Accept | 0.36538 | 0.10958 | 0.10154 | 0.12291 | svm | BoxConstraint: 271.6 | | | | | | | | | KernelScale: 2.743 |
| 57 | Accept | 0.16615 | 11.482 | 0.10154 | 0.12291 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 248 | | | | | | | | | MinLeafSize: 117 |
| 58 | Accept | 0.37179 | 0.095419 | 0.10154 | 0.12291 | svm | BoxConstraint: 7.5785 | | | | | | | | | KernelScale: 0.0066815 |
| 59 | Accept | 0.37179 | 0.097469 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.0017765 | | | | | | | | | KernelScale: 0.86786 |
| 60 | Accept | 0.37179 | 0.11284 | 0.10154 | 0.12291 | svm | BoxConstraint: 0.011465 | | | | | | | | | KernelScale: 0.02747 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 61 | Accept | 0.11692 | 0.077515 | 0.10154 | 0.12239 | tree | MinLeafSize: 12 |
| 62 | Accept | 0.29167 | 0.091617 | 0.10154 | 0.12239 | svm | BoxConstraint: 11.939 | | | | | | | | | KernelScale: 11.002 |
| 63 | Accept | 0.21795 | 0.067171 | 0.10154 | 0.12239 | knn | NumNeighbors: 6 |
| 64 | Accept | 0.18269 | 0.062887 | 0.10154 | 0.12239 | knn | NumNeighbors: 3 |
| 65 | Accept | 0.12923 | 0.075704 | 0.10154 | 0.11989 | tree | MinLeafSize: 3 |
| 66 | Accept | 0.16923 | 0.065889 | 0.10154 | 0.12048 | tree | MinLeafSize: 56 |
| 67 | Accept | 0.1891 | 0.068215 | 0.10154 | 0.12048 | knn | NumNeighbors: 1 |
| 68 | Accept | 0.13231 | 14.135 | 0.10154 | 0.12048 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 270 | | | | | | | | | MinLeafSize: 4 |
| 69 | Accept | 0.22769 | 0.060902 | 0.10154 | 0.12048 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 70 | Accept | 0.37231 | 0.24511 | 0.10154 | 0.12048 | nb | DistributionNames: kernel | | | | | | | | | Width: 1629.5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 71 | Accept | 0.16923 | 0.069135 | 0.10154 | 0.11947 | tree | MinLeafSize: 61 |
| 72 | Accept | 0.22769 | 0.060552 | 0.10154 | 0.11947 | nb | DistributionNames: normal | | | | | | | | | Width: NaN |
| 73 | Accept | 0.16308 | 10.133 | 0.10154 | 0.11947 | ensemble | Method: LogitBoost | | | | | | | | | NumLearningCycles: 217 | | | | | | | | | MinLeafSize: 70 |
| 74 | Accept | 0.13231 | 13.055 | 0.10154 | 0.11947 | ensemble | Method: Bag | | | | | | | | | NumLearningCycles: 257 | | | | | | | | | MinLeafSize: 2 |
| 75 | Accept | 0.21474 | 0.069312 | 0.10154 | 0.11947 | knn | NumNeighbors: 49 |
| 76 | Accept | 0.13846 | 0.083714 | 0.10154 | 0.1214 | tree | MinLeafSize: 25 |
| 77 | Accept | 0.12 | 0.069041 | 0.10154 | 0.11923 | tree | MinLeafSize: 6 |
| 78 | Accept | 0.10154 | 0.077399 | 0.10154 | 0.1118 | tree | MinLeafSize: 5 |
| 79 | Accept | 0.12 | 0.076269 | 0.10154 | 0.11007 | tree | MinLeafSize: 4 |
| 80 | Accept | 0.10154 | 0.09325 | 0.10154 | 0.10878 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 81 | Accept | 0.10154 | 0.074541 | 0.10154 | 0.10737 | tree | MinLeafSize: 5 |
| 82 | Accept | 0.12 | 0.069929 | 0.10154 | 0.1063 | tree | MinLeafSize: 4 |
| 83 | Accept | 0.10154 | 0.072591 | 0.10154 | 0.10514 | tree | MinLeafSize: 5 |
| 84 | Accept | 0.10154 | 0.071254 | 0.10154 | 0.10366 | tree | MinLeafSize: 5 |
| 85 | Accept | 0.10154 | 0.077378 | 0.10154 | 0.10361 | tree | MinLeafSize: 5 |
| 86 | Accept | 0.10154 | 0.070643 | 0.10154 | 0.10348 | tree | MinLeafSize: 5 |
| 87 | Accept | 0.12 | 0.070551 | 0.10154 | 0.10286 | tree | MinLeafSize: 4 |
| 88 | Accept | 0.12 | 0.078438 | 0.10154 | 0.1029 | tree | MinLeafSize: 6 |
| 89 | Accept | 0.10154 | 0.072996 | 0.10154 | 0.10262 | tree | MinLeafSize: 5 |
| 90 | Accept | 0.10154 | 0.078162 | 0.10154 | 0.10246 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 91 | Accept | 0.10154 | 0.07038 | 0.10154 | 0.10267 | tree | MinLeafSize: 5 |
| 92 | Accept | 0.10154 | 0.07752 | 0.10154 | 0.10257 | tree | MinLeafSize: 5 |
| 93 | Accept | 0.10154 | 0.076155 | 0.10154 | 0.10217 | tree | MinLeafSize: 5 |
| 94 | Accept | 0.10154 | 0.075983 | 0.10154 | 0.10221 | tree | MinLeafSize: 5 |
| 95 | Accept | 0.10154 | 0.07407 | 0.10154 | 0.10211 | tree | MinLeafSize: 5 |
| 96 | Accept | 0.10154 | 0.080633 | 0.10154 | 0.10207 | tree | MinLeafSize: 5 |
| 97 | Accept | 0.10154 | 0.086164 | 0.10154 | 0.10205 | tree | MinLeafSize: 5 |
| 98 | Accept | 0.10154 | 0.080264 | 0.10154 | 0.10191 | tree | MinLeafSize: 5 |
| 99 | Accept | 0.12308 | 0.076015 | 0.10154 | 0.1021 | tree | MinLeafSize: 17 |
| 100 | Accept | 0.10154 | 0.074579 | 0.10154 | 0.1019 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 101 | Accept | 0.10154 | 0.072077 | 0.10154 | 0.10186 | tree | MinLeafSize: 5 |
| 102 | Accept | 0.10154 | 0.071287 | 0.10154 | 0.10199 | tree | MinLeafSize: 5 |
| 103 | Accept | 0.10154 | 0.080003 | 0.10154 | 0.10186 | tree | MinLeafSize: 5 |
| 104 | Accept | 0.12 | 0.07462 | 0.10154 | 0.10189 | tree | MinLeafSize: 13 |
| 105 | Accept | 0.10154 | 0.077244 | 0.10154 | 0.10198 | tree | MinLeafSize: 5 |
| 106 | Accept | 0.12 | 0.080302 | 0.10154 | 0.10173 | tree | MinLeafSize: 4 |
| 107 | Accept | 0.10154 | 0.071858 | 0.10154 | 0.10183 | tree | MinLeafSize: 5 |
| 108 | Accept | 0.10154 | 0.076013 | 0.10154 | 0.10166 | tree | MinLeafSize: 5 |
| 109 | Accept | 0.10154 | 0.079106 | 0.10154 | 0.10164 | tree | MinLeafSize: 5 |
| 110 | Accept | 0.10154 | 0.075807 | 0.10154 | 0.10172 | tree | MinLeafSize: 5 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 111 | Accept | 0.10154 | 0.076903 | 0.10154 | 0.10159 | tree | MinLeafSize: 5 |
| 112 | Accept | 0.10154 | 0.077409 | 0.10154 | 0.10163 | tree | MinLeafSize: 5 |
| 113 | Accept | 0.10154 | 0.069569 | 0.10154 | 0.10165 | tree | MinLeafSize: 5 |
| 114 | Accept | 0.12308 | 0.076744 | 0.10154 | 0.10156 | tree | MinLeafSize: 18 |
| 115 | Accept | 0.19551 | 0.097443 | 0.10154 | 0.10156 | svm | BoxConstraint: 1.2977 | | | | | | | | | KernelScale: 33.654 |
| 116 | Accept | 0.26603 | 0.099474 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.082725 | | | | | | | | | KernelScale: 139.98 |
| 117 | Accept | 0.37179 | 0.077589 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.0065156 | | | | | | | | | KernelScale: 559.53 |
| 118 | Accept | 0.15385 | 0.08281 | 0.10154 | 0.10156 | svm | BoxConstraint: 8.5337 | | | | | | | | | KernelScale: 482.07 |
| 119 | Accept | 0.20833 | 0.074271 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.1729 | | | | | | | | | KernelScale: 980.38 |
| 120 | Accept | 0.17949 | 0.081488 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.8028 | | | | | | | | | KernelScale: 578.14 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 121 | Accept | 0.22115 | 0.076901 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.19345 | | | | | | | | | KernelScale: 367.75 |
| 122 | Accept | 0.17308 | 0.076171 | 0.10154 | 0.10156 | svm | BoxConstraint: 10.344 | | | | | | | | | KernelScale: 679.05 |
| 123 | Accept | 0.125 | 0.09187 | 0.10154 | 0.10156 | svm | BoxConstraint: 72.626 | | | | | | | | | KernelScale: 228.42 |
| 124 | Accept | 0.13462 | 0.15746 | 0.10154 | 0.10156 | svm | BoxConstraint: 586.5 | | | | | | | | | KernelScale: 176.02 |
| 125 | Accept | 0.13462 | 0.093366 | 0.10154 | 0.10156 | svm | BoxConstraint: 83.771 | | | | | | | | | KernelScale: 117.21 |
| 126 | Accept | 0.22436 | 0.10124 | 0.10154 | 0.10156 | svm | BoxConstraint: 962.52 | | | | | | | | | KernelScale: 20.898 |
| 127 | Accept | 0.15705 | 0.1133 | 0.10154 | 0.10156 | svm | BoxConstraint: 29.038 | | | | | | | | | KernelScale: 66.563 |
| 128 | Accept | 0.16346 | 0.10388 | 0.10154 | 0.10156 | svm | BoxConstraint: 156.8 | | | | | | | | | KernelScale: 62.775 |
| 129 | Accept | 0.13782 | 0.087555 | 0.10154 | 0.10156 | svm | BoxConstraint: 94.357 | | | | | | | | | KernelScale: 932.27 |
| 130 | Accept | 0.12821 | 0.093318 | 0.10154 | 0.10156 | svm | BoxConstraint: 25.982 | | | | | | | | | KernelScale: 247.06 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 131 | Accept | 0.13462 | 0.08648 | 0.10154 | 0.10156 | svm | BoxConstraint: 16.818 | | | | | | | | | KernelScale: 352.46 |
| 132 | Accept | 0.14103 | 0.090667 | 0.10154 | 0.10156 | svm | BoxConstraint: 15.817 | | | | | | | | | KernelScale: 130.15 |
| 133 | Accept | 0.12179 | 0.089208 | 0.10154 | 0.10156 | svm | BoxConstraint: 74.054 | | | | | | | | | KernelScale: 555.66 |
| 134 | Accept | 0.21474 | 0.082634 | 0.10154 | 0.10156 | svm | BoxConstraint: 0.80097 | | | | | | | | | KernelScale: 239.08 |
| 135 | Accept | 0.125 | 0.088947 | 0.10154 | 0.10156 | svm | BoxConstraint: 47.244 | | | | | | | | | KernelScale: 214.51 |
| 136 | Accept | 0.13141 | 0.07954 | 0.10154 | 0.10156 | svm | BoxConstraint: 13.389 | | | | | | | | | KernelScale: 145.55 |
| 137 | Accept | 0.13782 | 0.091167 | 0.10154 | 0.10156 | svm | BoxConstraint: 60.755 | | | | | | | | | KernelScale: 131.88 |
| 138 | Accept | 0.125 | 0.087338 | 0.10154 | 0.10156 | svm | BoxConstraint: 11.454 | | | | | | | | | KernelScale: 176.64 |
| 139 | Accept | 0.125 | 0.080567 | 0.10154 | 0.10156 | svm | BoxConstraint: 11.85 | | | | | | | | | KernelScale: 160.69 |
| 140 | Accept | 0.16667 | 0.083852 | 0.10154 | 0.10156 | svm | BoxConstraint: 11.155 | | | | | | | | | KernelScale: 652.28 |
|=================================================================================================================================| | Iter | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | result | loss | & validation (sec)| validation loss | validation loss | | | |=================================================================================================================================| | 141 | Accept | 0.125 | 0.089769 | 0.10154 | 0.10156 | svm | BoxConstraint: 10.933 | | | | | | | | | KernelScale: 172.97 |
| 142 | Accept | 0.21474 | 0.1003 | 0.10154 | 0.10156 | svm | BoxConstraint: 16.172 | | | | | | | | | KernelScale: 22.373 |
| 143 | Accept | 0.13782 | 0.0833 | 0.10154 | 0.10156 | svm | BoxConstraint: 7.6332 | | | | | | | | | KernelScale: 178.51 |
| 144 | Accept | 0.13462 | 0.10562 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.7646 | | | | | | | | | KernelScale: 160.72 |
| 145 | Accept | 0.14103 | 0.076631 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.8617 | | | | | | | | | KernelScale: 189.52 |
| 146 | Accept | 0.15385 | 0.082538 | 0.10154 | 0.10156 | svm | BoxConstraint: 6.0384 | | | | | | | | | KernelScale: 332.89 |
| 147 | Accept | 0.14423 | 0.077025 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.7255 | | | | | | | | | KernelScale: 182.19 |
| 148 | Accept | 0.13782 | 0.089308 | 0.10154 | 0.10156 | svm | BoxConstraint: 70.212 | | | | | | | | | KernelScale: 818.46 |
| 149 | Accept | 0.14103 | 0.08348 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.493 | | | | | | | | | KernelScale: 230.68 |
| 150 | Accept | 0.13782 | 0.079744 | 0.10154 | 0.10156 | svm | BoxConstraint: 5.3564 | | | | | | | | | KernelScale: 111.96 |
__________________________________________________________ Optimization completed. Total iterations: 150 Total elapsed time: 835.2167 seconds Total time for training and validation: 193.4979 seconds Best observed learner is a tree model with: MinLeafSize: 5 Observed validation loss: 0.10154 Time for training and validation: 0.079656 seconds Best estimated learner (returned model) is a tree model with: MinLeafSize: 5 Estimated validation loss: 0.10156 Estimated time for training and validation: 0.076205 seconds Documentation for fitcauto display
The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (carsTrain), the listed Learner (or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
Evaluate the performance of the model on the test set.
testAccuracy = 1 - loss(Mdl,carsTest,'Origin')testAccuracy = 0.9143
confusionchart(carsTest.Origin,predict(Mdl,carsTest))
Automatically Select Classifier Using Matrix Data
Use fitcauto to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in separate variables.
Load Data
Load the humanactivity data set. This data set contains 24,075 observations of five physical human activities: Sitting (1), Standing (2), Walking (3), Running (4), and Dancing (5). Each observation has 60 features extracted from acceleration data measured by smartphone accelerometer sensors. The variable feat contains the predictor data matrix of the 60 features for the 24,075 observations, and the response variable actid contains the activity IDs for the observations as integers.
load humanactivityPartition Data
Partition the data into training and test sets. Use 90% of the observations to select a model, and 10% of the observations to validate the final model returned by fitcauto. Use cvpartition to reserve 10% of the observations for testing.
rng('default') % For reproducibility of the partition c = cvpartition(actid,'Holdout',0.10); trainingIndices = training(c); % Indices for the training set XTrain = feat(trainingIndices,:); YTrain = actid(trainingIndices); testIndices = test(c); % Indices for the test set XTest = feat(testIndices,:); YTest = actid(testIndices);
Run fitcauto
Pass the training data to fitcauto. By default, fitcauto determines appropriate model (or learner) types to try, uses Bayesian optimization to find good hyperparameter values for those models, and returns a trained model with the best expected performance. Specify to run the optimization in parallel (requires Parallel Computing Toolbox™). Return a second output OptimizationResults that contains the details of the Bayesian optimization.
Expect this model selection process to take some time. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
options = struct('UseParallel',true); [Mdl,OptimizationResults] = fitcauto(XTrain,YTrain,'HyperparameterOptimizationOptions',options);
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Starting parallel pool (parpool) using the 'local' profile ... Connected to the parallel pool (number of workers: 6). Copying objective function to workers... Done copying objective function to workers.
Learner types to explore: ensemble, knn, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 150 Total time (MaxTime): Inf
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 1 | 6 | Best | 0.28088 | 48.752 | 0.28088 | 0.28088 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.22686 | | | | | | | | | | KernelScale: 330.4 |
| 2 | 6 | Best | 0.036459 | 51.455 | 0.036459 | 0.036459 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 254 | | | | | | | | | | MinLeafSize: 1786 | | | | | | | | | | MaxNumSplits: 12 |
| 3 | 6 | Best | 0.025845 | 5.1379 | 0.025845 | 0.025845 | tree | MinLeafSize: 59 |
| 4 | 6 | Best | 0.006415 | 60.999 | 0.006415 | 0.021738 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 214 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 23 |
| 5 | 6 | Accept | 0.025845 | 4.8823 | 0.006415 | 0.021738 | tree | MinLeafSize: 59 |
| 6 | 6 | Accept | 0.017768 | 5.4601 | 0.006415 | 0.021738 | tree | MinLeafSize: 9 |
| 7 | 6 | Accept | 0.050212 | 1.1024 | 0.006415 | 0.021738 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 8 | 6 | Accept | 0.050212 | 0.64183 | 0.006415 | 0.021738 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 9 | 6 | Accept | 0.019568 | 151.32 | 0.006415 | 0.021152 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 218 | | | | | | | | | | MinLeafSize: 2 | | | | | | | | | | MaxNumSplits: 63 |
| 10 | 6 | Accept | 0.026537 | 165.42 | 0.006415 | 0.022035 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 264 | | | | | | | | | | MinLeafSize: 7 | | | | | | | | | | MaxNumSplits: 36 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 11 | 6 | Accept | 0.59166 | 29.107 | 0.006415 | 0.022035 | nb | DistributionNames: kernel | | | | | | | | | | Width: 4.7212e-14 |
| 12 | 6 | Accept | 0.021645 | 50.626 | 0.006415 | 0.022122 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 243 | | | | | | | | | | MinLeafSize: 1247 | | | | | | | | | | MaxNumSplits: 45 |
| 13 | 6 | Accept | 0.043567 | 24.583 | 0.006415 | 0.022122 | knn | NumNeighbors: 144 |
| 14 | 6 | Accept | 0.028844 | 22.503 | 0.006415 | 0.022122 | knn | NumNeighbors: 18 |
| 15 | 6 | Accept | 0.04389 | 157.12 | 0.006415 | 0.022122 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.068467 | | | | | | | | | | KernelScale: 117.01 |
| 16 | 6 | Accept | 0.024598 | 20.721 | 0.006415 | 0.022122 | knn | NumNeighbors: 7 |
| 17 | 6 | Accept | 0.03009 | 20.969 | 0.006415 | 0.022122 | knn | NumNeighbors: 27 |
| 18 | 6 | Accept | 0.016753 | 6.5065 | 0.006415 | 0.021547 | tree | MinLeafSize: 2 |
| 19 | 6 | Accept | 0.040059 | 4.3496 | 0.006415 | 0.022122 | tree | MinLeafSize: 166 |
| 20 | 6 | Accept | 0.060319 | 2.2673 | 0.006415 | 0.022122 | tree | MinLeafSize: 1881 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 21 | 6 | Accept | 0.050212 | 0.97596 | 0.006415 | 0.022122 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 22 | 6 | Accept | 0.036552 | 20.775 | 0.006415 | 0.022122 | knn | NumNeighbors: 67 |
| 23 | 6 | Accept | 0.050212 | 0.55594 | 0.006415 | 0.022122 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 24 | 6 | Accept | 0.11076 | 36.229 | 0.006415 | 0.022122 | knn | NumNeighbors: 2637 |
| 25 | 6 | Accept | 0.27884 | 66.582 | 0.006415 | 0.024532 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 287 | | | | | | | | | | MinLeafSize: 4344 | | | | | | | | | | MaxNumSplits: 48 |
| 26 | 6 | Accept | 0.58127 | 31.861 | 0.006415 | 0.024532 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.6293e-06 |
| 27 | 6 | Accept | 0.01583 | 5.7511 | 0.006415 | 0.020656 | tree | MinLeafSize: 1 |
| 28 | 6 | Accept | 0.069319 | 1.806 | 0.006415 | 0.02077 | tree | MinLeafSize: 2284 |
| 29 | 6 | Accept | 0.59166 | 352.55 | 0.006415 | 0.02077 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 790.4 | | | | | | | | | | KernelScale: 0.014348 |
| 30 | 6 | Accept | 0.043336 | 3.7865 | 0.006415 | 0.020133 | tree | MinLeafSize: 432 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 31 | 6 | Accept | 0.10555 | 34.294 | 0.006415 | 0.020133 | knn | NumNeighbors: 2430 |
| 32 | 6 | Accept | 0.021276 | 4.582 | 0.006415 | 0.018661 | tree | MinLeafSize: 17 |
| 33 | 5 | Accept | 0.030829 | 159.57 | 0.006415 | 0.018642 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 288 | | | | | | | | | | MinLeafSize: 45 | | | | | | | | | | MaxNumSplits: 23 | | 34 | 5 | Accept | 0.014307 | 49.903 | 0.006415 | 0.018642 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | | MinLeafSize: 587 | | | | | | | | | | MaxNumSplits: 11 |
| 35 | 5 | Accept | 0.050212 | 1.025 | 0.006415 | 0.018642 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 36 | 6 | Accept | 0.74165 | 24.131 | 0.006415 | 0.018661 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 217 | | | | | | | | | | MinLeafSize: 8856 | | | | | | | | | | MaxNumSplits: 36 |
| 37 | 6 | Accept | 0.4226 | 558.84 | 0.006415 | 0.018661 | nb | DistributionNames: kernel | | | | | | | | | | Width: 72.906 |
| 38 | 6 | Accept | 0.57615 | 317.96 | 0.006415 | 0.018661 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.2347 | | | | | | | | | | KernelScale: 0.16176 |
| 39 | 6 | Accept | 0.59166 | 26.64 | 0.006415 | 0.018661 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1.191e-07 |
| 40 | 6 | Accept | 0.087087 | 30.271 | 0.006415 | 0.018661 | knn | NumNeighbors: 1634 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 41 | 6 | Accept | 0.73985 | 551.47 | 0.006415 | 0.018661 | nb | DistributionNames: kernel | | | | | | | | | | Width: 1055.8 |
| 42 | 4 | Accept | 0.025983 | 146.08 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | | MinLeafSize: 73 | | | | | | | | | | MaxNumSplits: 43 | | 43 | 4 | Accept | 0.02566 | 146.73 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 234 | | | | | | | | | | MinLeafSize: 73 | | | | | | | | | | MaxNumSplits: 43 | | 44 | 4 | Accept | 0.024922 | 124.65 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 206 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 39 |
| 45 | 6 | Accept | 0.025891 | 23.638 | 0.006415 | 0.018661 | knn | NumNeighbors: 6 |
| 46 | 5 | Accept | 0.025891 | 24.038 | 0.006415 | 0.018661 | knn | NumNeighbors: 6 | | 47 | 5 | Accept | 0.025891 | 23.646 | 0.006415 | 0.018661 | knn | NumNeighbors: 6 |
| 48 | 6 | Accept | 0.03009 | 188.48 | 0.006415 | 0.018661 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 299 | | | | | | | | | | MinLeafSize: 31 | | | | | | | | | | MaxNumSplits: 25 |
| 49 | 6 | Accept | 0.017214 | 6.0889 | 0.006415 | 0.017935 | tree | MinLeafSize: 4 |
| 50 | 6 | Accept | 0.01726 | 5.6027 | 0.006415 | 0.017303 | tree | MinLeafSize: 5 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 51 | 6 | Accept | 0.037244 | 158.2 | 0.006415 | 0.017303 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 5.9571 | | | | | | | | | | KernelScale: 840.87 |
| 52 | 6 | Accept | 0.046474 | 190.01 | 0.006415 | 0.017303 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 2.9119 | | | | | | | | | | KernelScale: 12.771 |
| 53 | 6 | Accept | 0.032398 | 155.47 | 0.006415 | 0.017303 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 253 | | | | | | | | | | MinLeafSize: 14 | | | | | | | | | | MaxNumSplits: 22 |
| 54 | 6 | Accept | 0.054135 | 3.0156 | 0.006415 | 0.017093 | tree | MinLeafSize: 783 |
| 55 | 6 | Accept | 0.049797 | 28.421 | 0.006415 | 0.017093 | knn | NumNeighbors: 331 |
| 56 | 6 | Accept | 0.046566 | 27.524 | 0.006415 | 0.017093 | knn | NumNeighbors: 193 |
| 57 | 6 | Accept | 0.36307 | 711.57 | 0.006415 | 0.017093 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.0011107 | | | | | | | | | | KernelScale: 0.80966 |
| 58 | 6 | Accept | 0.022706 | 23.417 | 0.006415 | 0.017093 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.3505 | | | | | | | | | | KernelScale: 127.55 |
| 59 | 6 | Accept | 0.028798 | 4.1232 | 0.006415 | 0.01733 | tree | MinLeafSize: 84 |
| 60 | 6 | Accept | 0.041351 | 28.037 | 0.006415 | 0.01733 | knn | NumNeighbors: 124 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 61 | 6 | Accept | 0.030044 | 26.265 | 0.006415 | 0.01733 | knn | NumNeighbors: 26 |
| 62 | 6 | Accept | 0.11838 | 284.42 | 0.006415 | 0.01733 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.0027874 | | | | | | | | | | KernelScale: 33.944 |
| 63 | 6 | Accept | 0.47116 | 54.038 | 0.006415 | 0.01733 | nb | DistributionNames: kernel | | | | | | | | | | Width: 4.7553e-05 |
| 64 | 6 | Accept | 0.26574 | 104.86 | 0.006415 | 0.01733 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0011441 |
| 65 | 6 | Accept | 0.050212 | 0.72243 | 0.006415 | 0.01733 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 66 | 6 | Accept | 0.01726 | 5.6362 | 0.006415 | 0.016846 | tree | MinLeafSize: 5 |
| 67 | 6 | Accept | 0.077072 | 268.17 | 0.006415 | 0.016846 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.026902 |
| 68 | 6 | Accept | 0.031613 | 25.909 | 0.006415 | 0.016846 | knn | NumNeighbors: 33 |
| 69 | 6 | Accept | 0.02003 | 92.379 | 0.006415 | 0.016846 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 609.47 | | | | | | | | | | KernelScale: 39.88 |
| 70 | 6 | Accept | 0.1145 | 92.795 | 0.006415 | 0.016846 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 292 | | | | | | | | | | MinLeafSize: 3870 | | | | | | | | | | MaxNumSplits: 33 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 71 | 6 | Accept | 0.012968 | 53.299 | 0.006415 | 0.016846 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 208 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 10 |
| 72 | 6 | Accept | 0.011999 | 53.54 | 0.006415 | 0.016846 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 215 | | | | | | | | | | MinLeafSize: 5 | | | | | | | | | | MaxNumSplits: 11 |
| 73 | 6 | Accept | 0.011999 | 50.887 | 0.006415 | 0.012076 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 205 | | | | | | | | | | MinLeafSize: 2 | | | | | | | | | | MaxNumSplits: 11 |
| 74 | 6 | Accept | 0.039921 | 40.808 | 0.006415 | 0.012733 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 200 | | | | | | | | | | MinLeafSize: 1921 | | | | | | | | | | MaxNumSplits: 11 |
| 75 | 6 | Accept | 0.04232 | 98.684 | 0.006415 | 0.012754 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 220 | | | | | | | | | | MinLeafSize: 1495 | | | | | | | | | | MaxNumSplits: 11 |
| 76 | 6 | Accept | 0.094702 | 397.88 | 0.006415 | 0.012754 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 671.76 | | | | | | | | | | KernelScale: 4.6418 |
| 77 | 6 | Accept | 0.065165 | 44.801 | 0.006415 | 0.013179 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 233 | | | | | | | | | | MinLeafSize: 2586 | | | | | | | | | | MaxNumSplits: 35 |
| 78 | 6 | Accept | 0.22503 | 164.64 | 0.006415 | 0.013179 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 57.71 | | | | | | | | | | KernelScale: 2.0632 |
| 79 | 6 | Accept | 0.03069 | 44.42 | 0.006415 | 0.013027 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 213 | | | | | | | | | | MinLeafSize: 1619 | | | | | | | | | | MaxNumSplits: 95 |
| 80 | 6 | Accept | 0.11459 | 71.56 | 0.006415 | 0.012064 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 212 | | | | | | | | | | MinLeafSize: 2669 | | | | | | | | | | MaxNumSplits: 31 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 81 | 6 | Accept | 0.044582 | 41.462 | 0.006415 | 0.013176 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 207 | | | | | | | | | | MinLeafSize: 2211 | | | | | | | | | | MaxNumSplits: 56 |
| 82 | 6 | Accept | 0.014722 | 223.7 | 0.006415 | 0.013176 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 832.84 | | | | | | | | | | KernelScale: 392.65 |
| 83 | 6 | Accept | 0.018322 | 23.34 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 322.92 | | | | | | | | | | KernelScale: 542.76 |
| 84 | 6 | Accept | 0.016984 | 28.833 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 727.84 | | | | | | | | | | KernelScale: 558.25 |
| 85 | 6 | Accept | 0.59166 | 2135.3 | 0.006415 | 0.013176 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.045413 | | | | | | | | | | KernelScale: 0.0034709 |
| 86 | 6 | Accept | 0.012461 | 39.005 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 806.52 | | | | | | | | | | KernelScale: 244.64 |
| 87 | 6 | Accept | 0.016753 | 22.649 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 173.38 | | | | | | | | | | KernelScale: 329.41 |
| 88 | 6 | Accept | 0.016845 | 28.292 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 242.37 | | | | | | | | | | KernelScale: 62.644 |
| 89 | 6 | Accept | 0.016799 | 20.771 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 80.674 | | | | | | | | | | KernelScale: 242.97 |
| 90 | 6 | Accept | 0.041259 | 175.27 | 0.006415 | 0.013176 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 2.0147 | | | | | | | | | | KernelScale: 549.95 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 91 | 6 | Accept | 0.017953 | 19.778 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 63.367 | | | | | | | | | | KernelScale: 263.72 |
| 92 | 6 | Accept | 0.019568 | 19.245 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 55.612 | | | | | | | | | | KernelScale: 344.11 |
| 93 | 6 | Accept | 0.016061 | 19.511 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 60.019 | | | | | | | | | | KernelScale: 185.49 |
| 94 | 6 | Accept | 0.11847 | 149.5 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 770.37 | | | | | | | | | | KernelScale: 3.1188 |
| 95 | 6 | Accept | 0.017953 | 18.428 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 43.94 | | | | | | | | | | KernelScale: 234.89 |
| 96 | 6 | Accept | 0.025106 | 25.204 | 0.006415 | 0.013176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.1294 | | | | | | | | | | KernelScale: 237.71 |
| 97 | 6 | Accept | 0.011676 | 70.358 | 0.006415 | 0.012446 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 283 | | | | | | | | | | MinLeafSize: 4 | | | | | | | | | | MaxNumSplits: 10 |
| 98 | 6 | Accept | 0.031983 | 37.179 | 0.006415 | 0.012446 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.832 | | | | | | | | | | KernelScale: 449.73 |
| 99 | 6 | Accept | 0.0097379 | 76.639 | 0.006415 | 0.011402 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 299 | | | | | | | | | | MinLeafSize: 4 | | | | | | | | | | MaxNumSplits: 12 |
| 100 | 6 | Accept | 0.22416 | 684.65 | 0.006415 | 0.011402 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 632.44 | | | | | | | | | | KernelScale: 1.8647 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 101 | 6 | Accept | 0.11478 | 87.263 | 0.006415 | 0.011913 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 272 | | | | | | | | | | MinLeafSize: 3176 | | | | | | | | | | MaxNumSplits: 54 |
| 102 | 6 | Accept | 0.0081687 | 75.177 | 0.006415 | 0.01045 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 291 | | | | | | | | | | MinLeafSize: 6 | | | | | | | | | | MaxNumSplits: 14 |
| 103 | 6 | Accept | 0.010753 | 70.01 | 0.006415 | 0.010258 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 284 | | | | | | | | | | MinLeafSize: 1 | | | | | | | | | | MaxNumSplits: 11 |
| 104 | 6 | Accept | 0.36076 | 77 | 0.006415 | 0.010258 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0054235 | | | | | | | | | | KernelScale: 149.67 |
| 105 | 6 | Accept | 0.0084456 | 62.748 | 0.006415 | 0.0092051 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 214 | | | | | | | | | | MinLeafSize: 3 | | | | | | | | | | MaxNumSplits: 16 |
| 106 | 6 | Accept | 0.012738 | 32.34 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 529.51 | | | | | | | | | | KernelScale: 161.83 |
| 107 | 6 | Accept | 0.031521 | 41.482 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.6738 | | | | | | | | | | KernelScale: 289.2 |
| 108 | 6 | Accept | 0.021368 | 38.604 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 567.35 | | | | | | | | | | KernelScale: 36.573 |
| 109 | 6 | Accept | 0.050212 | 0.59557 | 0.006415 | 0.0092051 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 110 | 6 | Accept | 0.57883 | 127.15 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0011506 | | | | | | | | | | KernelScale: 211.98 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 111 | 6 | Accept | 0.050212 | 0.685 | 0.006415 | 0.0092051 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 112 | 6 | Accept | 0.029583 | 33.504 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.49417 | | | | | | | | | | KernelScale: 209.72 |
| 113 | 6 | Accept | 0.012138 | 41.159 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 967.85 | | | | | | | | | | KernelScale: 153.85 |
| 114 | 6 | Accept | 0.030921 | 38.482 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.57771 | | | | | | | | | | KernelScale: 262.15 |
| 115 | 6 | Accept | 0.1295 | 52.128 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.35331 | | | | | | | | | | KernelScale: 329.77 |
| 116 | 6 | Accept | 0.028475 | 33.553 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.37306 | | | | | | | | | | KernelScale: 157.01 |
| 117 | 6 | Accept | 0.050212 | 0.62316 | 0.006415 | 0.0092051 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 118 | 6 | Accept | 0.030737 | 37 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.37996 | | | | | | | | | | KernelScale: 204.24 |
| 119 | 6 | Accept | 0.021229 | 36.97 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 803.51 | | | | | | | | | | KernelScale: 38.271 |
| 120 | 6 | Accept | 0.015184 | 24.004 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 119.72 | | | | | | | | | | KernelScale: 145.99 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 121 | 6 | Accept | 0.06955 | 101.38 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 39.152 | | | | | | | | | | KernelScale: 8.5572 |
| 122 | 6 | Accept | 0.028291 | 32.637 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.32264 | | | | | | | | | | KernelScale: 134.73 |
| 123 | 6 | Accept | 0.041951 | 51.65 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.28203 | | | | | | | | | | KernelScale: 237.1 |
| 124 | 6 | Accept | 0.039921 | 216.38 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.049329 | | | | | | | | | | KernelScale: 190.24 |
| 125 | 6 | Accept | 0.048828 | 91.888 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 180.12 | | | | | | | | | | KernelScale: 12.235 |
| 126 | 6 | Accept | 0.030644 | 38.563 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.25725 | | | | | | | | | | KernelScale: 171.19 |
| 127 | 6 | Accept | 0.029537 | 37.409 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.25269 | | | | | | | | | | KernelScale: 145.85 |
| 128 | 6 | Accept | 0.02949 | 31.222 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.24034 | | | | | | | | | | KernelScale: 59.755 |
| 129 | 6 | Accept | 0.029537 | 130.19 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 76.533 | | | | | | | | | | KernelScale: 22.982 |
| 130 | 6 | Accept | 0.094702 | 174.14 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.043666 | | | | | | | | | | KernelScale: 25.411 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 131 | 6 | Accept | 0.037382 | 46.207 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.23867 | | | | | | | | | | KernelScale: 204.62 |
| 132 | 6 | Accept | 0.022152 | 40.588 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 887.62 | | | | | | | | | | KernelScale: 32.987 |
| 133 | 6 | Accept | 0.032629 | 32.651 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.22511 | | | | | | | | | | KernelScale: 52.294 |
| 134 | 6 | Accept | 0.02806 | 131.12 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.20346 | | | | | | | | | | KernelScale: 122.85 |
| 135 | 6 | Accept | 0.59166 | 3480 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 658.37 | | | | | | | | | | KernelScale: 0.0016161 |
| 136 | 6 | Accept | 0.59166 | 3481.3 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 658.37 | | | | | | | | | | KernelScale: 0.0016161 |
| 137 | 6 | Accept | 0.032121 | 44.172 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.20648 | | | | | | | | | | KernelScale: 162.33 |
| 138 | 6 | Accept | 0.30755 | 71.9 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.015886 | | | | | | | | | | KernelScale: 120.23 |
| 139 | 6 | Accept | 0.040936 | 233.89 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.046614 | | | | | | | | | | KernelScale: 213.78 |
| 140 | 5 | Accept | 0.040521 | 224.72 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.043998 | | | | | | | | | | KernelScale: 195.58 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 141 | 5 | Accept | 0.22208 | 64.659 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.049755 | | | | | | | | | | KernelScale: 151.01 |
| 142 | 6 | Accept | 0.029444 | 30.014 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.21789 | | | | | | | | | | KernelScale: 64.023 |
| 143 | 6 | Accept | 0.046013 | 49.025 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.14805 | | | | | | | | | | KernelScale: 178.98 |
| 144 | 6 | Accept | 0.013291 | 42.726 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 960.14 | | | | | | | | | | KernelScale: 334.83 |
| 145 | 6 | Accept | 0.59166 | 3493.5 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 17.982 | | | | | | | | | | KernelScale: 0.0043756 |
| 146 | 6 | Accept | 0.089533 | 189.71 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.2326 | | | | | | | | | | KernelScale: 15.411 |
| 147 | 6 | Accept | 0.03369 | 49.586 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.21482 | | | | | | | | | | KernelScale: 180.5 |
| 148 | 6 | Accept | 0.013384 | 42.876 | 0.006415 | 0.0092051 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 920.16 | | | | | | | | | | KernelScale: 334.75 |
| 149 | 6 | Accept | 0.027368 | 117.6 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.22783 | | | | | | | | | | KernelScale: 105.59 |
| 150 | 6 | Accept | 0.041674 | 234.61 | 0.006415 | 0.0092051 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.032335 | | | | | | | | | | KernelScale: 182.17 |
__________________________________________________________ Optimization completed. Total iterations: 150 Total elapsed time: 4534.2478 seconds Total time for training and validation: 24883.8563 seconds Best observed learner is an ensemble model with: Method: AdaBoostM2 NumLearningCycles: 214 MinLeafSize: 5 MaxNumSplits: 23 Observed validation loss: 0.006415 Time for training and validation: 60.9987 seconds Best estimated learner (returned model) is an ensemble model with: Method: AdaBoostM2 NumLearningCycles: 214 MinLeafSize: 3 MaxNumSplits: 16 Estimated validation loss: 0.0092051 Estimated time for training and validation: 57.8146 seconds Documentation for fitcauto display
The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (XTrain and YTrain), the listed Learner (or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
Evaluate the final model performance on the test data set.
testAccuracy = 1 - loss(Mdl,XTest,YTest)
testAccuracy = 0.9917
The final model correctly classifies over 99% of the observations.
Combine Feature Selection and Automated Classifier Selection
Use fitcauto to automatically select a classification model with optimized hyperparameters, given predictor and response data stored in a table. Before passing data to fitcauto, perform feature selection to remove unimportant predictors from the data set.
Load and Partition Data
Read the sample file CreditRating_Historical.dat into a table. The predictor data consists of financial ratios and industry sector information for a list of corporate customers. The response variable consists of credit ratings assigned by a rating agency. Preview the first few rows of the data set.
creditrating = readtable('CreditRating_Historical.dat');
head(creditrating)ans=8×8 table
ID WC_TA RE_TA EBIT_TA MVE_BVTD S_TA Industry Rating
_____ ______ ______ _______ ________ _____ ________ _______
62394 0.013 0.104 0.036 0.447 0.142 3 {'BB' }
48608 0.232 0.335 0.062 1.969 0.281 8 {'A' }
42444 0.311 0.367 0.074 1.935 0.366 1 {'A' }
48631 0.194 0.263 0.062 1.017 0.228 4 {'BBB'}
43768 0.121 0.413 0.057 3.647 0.466 12 {'AAA'}
39255 -0.117 -0.799 0.01 0.179 0.082 4 {'CCC'}
62236 0.087 0.158 0.049 0.816 0.324 2 {'BBB'}
39354 0.005 0.181 0.034 2.597 0.388 7 {'AA' }
Because each value in the ID variable is a unique customer ID, that is, length(unique(creditrating.ID)) is equal to the number of observations in creditrating, the ID variable is a poor predictor. Remove the ID variable from the table, and convert the Industry variable to a categorical variable.
creditrating = removevars(creditrating,'ID');
creditrating.Industry = categorical(creditrating.Industry);Partition the data into training and test sets. Use approximately 85% of the observations for the model selection and hyperparameter tuning process, and 15% of the observations to test the performance of the final model returned by fitcauto on new data. Use cvpartition to partition the data.
rng('default') % For reproducibility of the partition c = cvpartition(creditrating.Rating,'Holdout',0.15); trainingIndices = training(c); % Indices for the training set testIndices = test(c); % Indices for the test set creditTrain = creditrating(trainingIndices,:); creditTest = creditrating(testIndices,:);
Perform Feature Selection
Before passing the training data to fitcauto, find the important predictors by using the fscchi2 function. Visualize the predictor scores by using the bar function. Because some scores can be Inf, and bar discards Inf values, plot the finite scores first and then plot a finite representation of the Inf scores in a different color.
[idx,scores] = fscchi2(creditTrain,'Rating'); bar(scores(idx)) % Represents finite scores hold on veryImportant = isinf(scores); finiteMax = max(scores(~veryImportant)); bar(finiteMax*veryImportant(idx)) % Represents Inf scores hold off xticklabels(strrep(creditTrain.Properties.VariableNames(idx),'_','\_')) xtickangle(45) legend({'Finite Scores','Inf Scores'})
Notice that the Industry predictor has a low score corresponding to a p-value that is greater than 0.05, which indicates that Industry might not be an important feature. Remove the Industry feature from the training and test data sets.
creditTrain = removevars(creditTrain,'Industry'); creditTest = removevars(creditTest,'Industry');
Run fitcauto
Pass the training data to fitcauto. The function uses Bayesian optimization to select models and their hyperparameter values, and returns a trained model Mdl with the best expected performance. Specify to try all available learner types and run the optimization in parallel (requires Parallel Computing Toolbox™). Return a second output Results that contains the details of the Bayesian optimization.
Expect this process to take some time. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.
options = struct('UseParallel',true); [Mdl,Results] = fitcauto(creditTrain,'Rating', ... 'Learners','all','HyperparameterOptimizationOptions',options);
Warning: It is recommended that you first standardize all numeric predictors when optimizing the Naive Bayes 'Width' parameter. Ignore this warning if you have done that.
Starting parallel pool (parpool) using the 'local' profile ... Connected to the parallel pool (number of workers: 6).
Copying objective function to workers... Done copying objective function to workers.
Learner types to explore: discr, ensemble, kernel, knn, linear, nb, svm, tree Total iterations (MaxObjectiveEvaluations): 240 Total time (MaxTime): Inf
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 1 | 6 | Best | 0.42716 | 3.0379 | 0.42716 | 0.42716 | discr | Delta: 0.00046441 | | | | | | | | | | Gamma: 0.2485 |
| 2 | 4 | Accept | 0.74185 | 4.7899 | 0.24948 | 0.29794 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.48455 | | | | | | | | | | KernelScale: 354.44 | | 3 | 4 | Best | 0.24948 | 5.0813 | 0.24948 | 0.29794 | linear | Coding: onevsone | | | | | | | | | | Lambda: 6.3551e-08 | | | | | | | | | | Learner: logistic | | 4 | 4 | Accept | 0.29794 | 3.7295 | 0.24948 | 0.29794 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 12 | | | | | | | | | | LearnRate: 0.063776 | | | | | | | | | | MinLeafSize: 277 |
| 5 | 3 | Accept | 0.25097 | 9.2655 | 0.24948 | 0.25067 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 7.8433 | | | | | | | | | | Lambda: 1.4468e-06 | | 6 | 3 | Accept | 0.25067 | 0.81139 | 0.24948 | 0.25067 | knn | NumNeighbors: 105 | | | | | | | | | | Distance: minkowski |
| 7 | 6 | Accept | 0.52917 | 2.3362 | 0.24948 | 0.25067 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.002417 | | | | | | | | | | KernelScale: 356.9 |
| 8 | 3 | Accept | 0.55818 | 0.63908 | 0.24948 | 0.25067 | discr | Delta: 0.98612 | | | | | | | | | | Gamma: 0.86519 | | 9 | 3 | Accept | 0.3781 | 1.6777 | 0.24948 | 0.25067 | linear | Coding: onevsall | | | | | | | | | | Lambda: 1.0412e-06 | | | | | | | | | | Learner: logistic | | 10 | 3 | Accept | 0.43225 | 0.80766 | 0.24948 | 0.25067 | discr | Delta: 0.00013711 | | | | | | | | | | Gamma: 0.60585 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 11 | 3 | Accept | 0.47712 | 3.3756 | 0.24948 | 0.25067 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 2.7347 | | | | | | | | | | KernelScale: 24.465 |
| 12 | 6 | Accept | 0.25695 | 2.3709 | 0.24948 | 0.25067 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.057566 |
| 13 | 4 | Accept | 0.26413 | 0.49941 | 0.24379 | 0.25067 | tree | MinLeafSize: 30 | | 14 | 4 | Accept | 0.42327 | 0.85101 | 0.24379 | 0.25067 | knn | NumNeighbors: 56 | | | | | | | | | | Distance: cosine | | 15 | 4 | Best | 0.24379 | 1.8084 | 0.24379 | 0.25067 | linear | Coding: onevsone | | | | | | | | | | Lambda: 5.9172e-05 | | | | | | | | | | Learner: svm |
| 16 | 3 | Accept | 0.81544 | 4.9586 | 0.24379 | 0.25067 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0043375 | | | | | | | | | | Lambda: 0.0023789 | | 17 | 3 | Accept | 0.45169 | 0.96723 | 0.24379 | 0.25067 | linear | Coding: onevsall | | | | | | | | | | Lambda: 0.0028505 | | | | | | | | | | Learner: svm |
| 18 | 6 | Accept | 0.33712 | 0.19972 | 0.24379 | 0.25695 | knn | NumNeighbors: 1 | | | | | | | | | | Distance: cityblock |
| 19 | 3 | Accept | 0.4834 | 0.38951 | 0.24379 | 0.25695 | knn | NumNeighbors: 72 | | | | | | | | | | Distance: correlation | | 20 | 3 | Accept | 0.46336 | 0.78881 | 0.24379 | 0.25695 | linear | Coding: onevsall | | | | | | | | | | Lambda: 0.0075732 | | | | | | | | | | Learner: svm | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 21 | 3 | Accept | 0.82082 | 2.9223 | 0.24379 | 0.25695 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0042587 | | | | | | | | | | Lambda: 0.0014754 | | 22 | 3 | Accept | 0.61292 | 1.8557 | 0.24379 | 0.25695 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 13 | | | | | | | | | | LearnRate: 0.055349 | | | | | | | | | | MinLeafSize: 910 |
| 23 | 6 | Accept | 0.43255 | 0.68689 | 0.24379 | 0.25695 | discr | Delta: 0.016844 | | | | | | | | | | Gamma: 0.64466 |
| 24 | 4 | Accept | 0.28866 | 1.9017 | 0.24379 | 0.25695 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 10 | | | | | | | | | | LearnRate: 0.11662 | | | | | | | | | | MinLeafSize: 181 | | 25 | 4 | Accept | 0.74185 | 1.3546 | 0.24379 | 0.25695 | knn | NumNeighbors: 1314 | | | | | | | | | | Distance: hamming | | 26 | 4 | Accept | 0.42746 | 0.69002 | 0.24379 | 0.25695 | discr | Delta: 2.2544e-06 | | | | | | | | | | Gamma: 0.87275 |
| 27 | 3 | Accept | 0.25606 | 12.143 | 0.24379 | 0.25498 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 132 | | | | | | | | | | LearnRate: 0.92674 | | | | | | | | | | MinLeafSize: 127 | | 28 | 3 | Accept | 0.25366 | 2.4732 | 0.24379 | 0.25498 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.10033 |
| 29 | 6 | Accept | 0.66796 | 0.22938 | 0.24379 | 0.25498 | knn | NumNeighbors: 77 | | | | | | | | | | Distance: jaccard |
| 30 | 4 | Accept | 0.69488 | 1.9352 | 0.242 | 0.25498 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 8.4886 | | | | | | | | | | KernelScale: 192.19 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 31 | 4 | Best | 0.242 | 1.9467 | 0.242 | 0.25498 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 5.425 | | | | | | | | | | KernelScale: 2.434 | | 32 | 4 | Accept | 0.32306 | 0.48104 | 0.242 | 0.25498 | tree | MinLeafSize: 2 |
| 33 | 4 | Accept | 0.43225 | 0.10463 | 0.242 | 0.25498 | discr | Delta: 0.00015292 | | | | | | | | | | Gamma: 0.51045 |
| 34 | 4 | Accept | 0.32994 | 0.2065 | 0.242 | 0.25498 | tree | MinLeafSize: 3 |
| 35 | 6 | Accept | 0.53814 | 2.8748 | 0.242 | 0.25498 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.8148 | | | | | | | | | | KernelScale: 382.11 |
| 36 | 4 | Accept | 0.24529 | 105.62 | 0.242 | 0.25498 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.25488 | | | | | | | | | | KernelScale: 0.0037823 | | 37 | 4 | Accept | 0.53814 | 3.3272 | 0.242 | 0.25498 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.8148 | | | | | | | | | | KernelScale: 382.11 | | 38 | 4 | Accept | 0.53814 | 3.8593 | 0.242 | 0.25498 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.8148 | | | | | | | | | | KernelScale: 382.11 |
| 39 | 4 | Accept | 0.25965 | 14.211 | 0.242 | 0.25498 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 150 | | | | | | | | | | LearnRate: 0.014842 | | | | | | | | | | MinLeafSize: 21 |
| 40 | 5 | Accept | 0.42656 | 0.16731 | 0.242 | 0.25498 | discr | Delta: 0.0020866 | | | | | | | | | | Gamma: 0.091054 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 41 | 5 | Accept | 0.42656 | 0.11476 | 0.242 | 0.25498 | discr | Delta: 0.0020866 | | | | | | | | | | Gamma: 0.091054 |
| 42 | 4 | Accept | 0.2767 | 21.389 | 0.242 | 0.25498 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 221 | | | | | | | | | | LearnRate: 0.0028588 | | | | | | | | | | MinLeafSize: 1 | | 43 | 4 | Accept | 0.29973 | 0.1848 | 0.242 | 0.25498 | tree | MinLeafSize: 7 |
| 44 | 4 | Accept | 0.25935 | 20.084 | 0.242 | 0.25498 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 304 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 100 |
| 45 | 3 | Accept | 0.24499 | 6.5071 | 0.242 | 0.26328 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.019387 | | | | | | | | | | KernelScale: 0.0047515 | | 46 | 3 | Accept | 0.28059 | 0.19378 | 0.242 | 0.26328 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 47 | 6 | Accept | 0.27281 | 0.13181 | 0.242 | 0.26328 | knn | NumNeighbors: 8 | | | | | | | | | | Distance: chebychev |
| 48 | 3 | Accept | 0.2429 | 1.5474 | 0.242 | 0.26328 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.16719 | | | | | | | | | | KernelScale: 0.11257 | | 49 | 3 | Accept | 0.2423 | 1.5219 | 0.242 | 0.26328 | linear | Coding: onevsone | | | | | | | | | | Lambda: 4.28e-07 | | | | | | | | | | Learner: svm | | 50 | 3 | Accept | 0.67125 | 0.68464 | 0.242 | 0.26328 | knn | NumNeighbors: 86 | | | | | | | | | | Distance: jaccard | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 51 | 3 | Accept | 0.46485 | 0.8417 | 0.242 | 0.26328 | tree | MinLeafSize: 645 |
| 52 | 6 | Accept | 0.63147 | 3.1926 | 0.242 | 0.26328 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 176.2 | | | | | | | | | | Lambda: 2.3903e-06 |
| 53 | 4 | Accept | 0.37571 | 0.11086 | 0.242 | 0.26706 | tree | MinLeafSize: 473 | | 54 | 4 | Accept | 0.29136 | 0.43908 | 0.242 | 0.26706 | knn | NumNeighbors: 354 | | | | | | | | | | Distance: euclidean | | 55 | 4 | Accept | 0.28059 | 0.50642 | 0.242 | 0.26706 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 56 | 3 | Accept | 0.36375 | 5.2617 | 0.242 | 0.26706 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 28.598 | | | | | | | | | | Lambda: 8.3238e-05 | | 57 | 3 | Accept | 0.27251 | 0.13425 | 0.242 | 0.26706 | tree | MinLeafSize: 20 |
| 58 | 6 | Accept | 0.43225 | 0.083255 | 0.242 | 0.26706 | discr | Delta: 0.021467 | | | | | | | | | | Gamma: 0.66016 |
| 59 | 3 | Accept | 0.28059 | 0.10921 | 0.242 | 0.26106 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | 60 | 3 | Accept | 0.42537 | 0.15885 | 0.242 | 0.26106 | discr | Delta: 0.001728 | | | | | | | | | | Gamma: 0.89471 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 61 | 3 | Accept | 0.81484 | 2.9209 | 0.242 | 0.26106 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0063987 | | | | | | | | | | Lambda: 0.0075855 | | 62 | 3 | Accept | 0.24948 | 2.4493 | 0.242 | 0.26106 | linear | Coding: onevsone | | | | | | | | | | Lambda: 1.3237e-07 | | | | | | | | | | Learner: logistic |
| 63 | 6 | Accept | 0.68531 | 0.40944 | 0.242 | 0.26106 | knn | NumNeighbors: 260 | | | | | | | | | | Distance: jaccard |
| 64 | 3 | Accept | 0.32426 | 9.8071 | 0.242 | 0.26007 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 132 | | | | | | | | | | LearnRate: 0.0014516 | | | | | | | | | | MinLeafSize: 104 | | 65 | 3 | Accept | 0.55369 | 0.69919 | 0.242 | 0.26007 | knn | NumNeighbors: 615 | | | | | | | | | | Distance: correlation | | 66 | 3 | Accept | 0.24319 | 1.5742 | 0.242 | 0.26007 | linear | Coding: onevsone | | | | | | | | | | Lambda: 4.3001e-09 | | | | | | | | | | Learner: svm | | 67 | 3 | Accept | 0.70894 | 1.9822 | 0.242 | 0.26007 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45564 | | | | | | | | | | KernelScale: 51.195 |
| 68 | 6 | Accept | 0.46485 | 0.17082 | 0.242 | 0.26007 | tree | MinLeafSize: 611 |
| 69 | 5 | Accept | 0.74185 | 2.1172 | 0.242 | 0.26007 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.94197 | | | | | | | | | | KernelScale: 524.56 | | 70 | 5 | Accept | 0.83548 | 6.3585 | 0.242 | 0.26007 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.0038762 | | | | | | | | | | Lambda: 2.288e-06 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 71 | 3 | Accept | 0.25905 | 16.096 | 0.242 | 0.26007 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 160 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 2 | | 72 | 3 | Accept | 0.28448 | 13.826 | 0.242 | 0.26007 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 161 | | | | | | | | | | LearnRate: 0.7581 | | | | | | | | | | MinLeafSize: 14 | | 73 | 3 | Accept | 0.25695 | 0.24642 | 0.242 | 0.26007 | knn | NumNeighbors: 14 | | | | | | | | | | Distance: cityblock |
| 74 | 6 | Accept | 0.32396 | 1.8799 | 0.242 | 0.26007 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.86904 | | | | | | | | | | Lambda: 0.29724 |
| 75 | 4 | Accept | 0.32456 | 0.20481 | 0.242 | 0.26007 | tree | MinLeafSize: 5 | | 76 | 4 | Accept | 0.32994 | 0.35799 | 0.242 | 0.26007 | tree | MinLeafSize: 3 | | 77 | 4 | Accept | 0.26054 | 4.1734 | 0.242 | 0.26007 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 32 | | | | | | | | | | LearnRate: 0.06853 | | | | | | | | | | MinLeafSize: 19 |
| 78 | 4 | Accept | 0.43703 | 0.92964 | 0.242 | 0.24831 | linear | Coding: onevsall | | | | | | | | | | Lambda: 0.013265 | | | | | | | | | | Learner: logistic |
| 79 | 4 | Accept | 0.31588 | 17.116 | 0.242 | 0.24831 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 201 | | | | | | | | | | LearnRate: 0.0012955 | | | | | | | | | | MinLeafSize: 319 |
| 80 | 3 | Accept | 0.25277 | 7.8173 | 0.242 | 0.24831 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.78697 | | | | | | | | | | Lambda: 4.1197e-06 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 81 | 3 | Accept | 0.43015 | 0.10239 | 0.242 | 0.24831 | discr | Delta: 0.0069822 | | | | | | | | | | Gamma: 0.49526 |
| 82 | 6 | Accept | 0.42208 | 0.096953 | 0.242 | 0.24831 | discr | Delta: 0.057485 | | | | | | | | | | Gamma: 0.045714 |
| 83 | 3 | Accept | 0.52617 | 2.3915 | 0.242 | 0.24831 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.26869 | | | | | | | | | | KernelScale: 17.595 | | 84 | 3 | Accept | 0.43344 | 1.2875 | 0.242 | 0.24831 | knn | NumNeighbors: 119 | | | | | | | | | | Distance: mahalanobis | | 85 | 3 | Accept | 0.30093 | 1.3003 | 0.242 | 0.24831 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.047624 | | | | | | | | | | Learner: svm | | 86 | 3 | Accept | 0.42267 | 0.64506 | 0.242 | 0.24831 | knn | NumNeighbors: 48 | | | | | | | | | | Distance: cosine |
| 87 | 6 | Accept | 0.32905 | 0.26891 | 0.242 | 0.24831 | knn | NumNeighbors: 65 | | | | | | | | | | Distance: seuclidean |
| 88 | 4 | Accept | 0.24349 | 1.9282 | 0.242 | 0.24684 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0024196 | | | | | | | | | | KernelScale: 0.0082547 | | 89 | 4 | Accept | 0.24499 | 1.6543 | 0.242 | 0.24684 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.3697e-06 | | | | | | | | | | Learner: svm | | 90 | 4 | Accept | 0.24469 | 2.1463 | 0.242 | 0.24684 | linear | Coding: onevsone | | | | | | | | | | Lambda: 6.5777e-05 | | | | | | | | | | Learner: logistic |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 91 | 4 | Accept | 0.28059 | 0.15481 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 92 | 5 | Accept | 0.28059 | 0.38138 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | 93 | 5 | Accept | 0.28059 | 0.15804 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 94 | 4 | Accept | 0.29674 | 11.234 | 0.242 | 0.24684 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 147 | | | | | | | | | | LearnRate: 0.95321 | | | | | | | | | | MinLeafSize: 112 | | 95 | 4 | Accept | 0.28059 | 0.12279 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 96 | 4 | Accept | 0.29704 | 1.1016 | 0.242 | 0.24684 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 11 | | | | | | | | | | LearnRate: 0.0072731 | | | | | | | | | | MinLeafSize: 12 |
| 97 | 4 | Accept | 0.28059 | 0.099833 | 0.242 | 0.24684 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 98 | 4 | Best | 0.2411 | 1.3634 | 0.2411 | 0.24471 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.00056045 | | | | | | | | | | Learner: svm |
| 99 | 4 | Accept | 0.74185 | 0.10441 | 0.2411 | 0.24471 | discr | Delta: 39.281 | | | | | | | | | | Gamma: 0.77032 |
| 100 | 4 | Accept | 0.30093 | 0.11043 | 0.2411 | 0.24471 | tree | MinLeafSize: 135 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 101 | 4 | Accept | 0.46844 | 0.17182 | 0.2411 | 0.24471 | knn | NumNeighbors: 2 | | | | | | | | | | Distance: cosine |
| 102 | 4 | Accept | 0.25426 | 3.1634 | 0.2411 | 0.24471 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 38 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 28 |
| 103 | 4 | Accept | 0.24469 | 1.6558 | 0.2411 | 0.24558 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.0012413 | | | | | | | | | | Learner: logistic |
| 104 | 4 | Accept | 0.43823 | 3.6572 | 0.2411 | 0.24558 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 13.293 | | | | | | | | | | Lambda: 0.0031304 |
| 105 | 4 | Accept | 0.31977 | 1.3423 | 0.2411 | 0.24556 | linear | Coding: onevsone | | | | | | | | | | Lambda: 0.040102 | | | | | | | | | | Learner: logistic |
| 106 | 6 | Accept | 0.24678 | 2.0849 | 0.2411 | 0.24529 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.9918e-05 | | | | | | | | | | Learner: logistic |
| 107 | 5 | Accept | 0.24678 | 1.9627 | 0.2411 | 0.24441 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.9918e-05 | | | | | | | | | | Learner: logistic | | 108 | 5 | Accept | 0.24678 | 2.3413 | 0.2411 | 0.24441 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.9918e-05 | | | | | | | | | | Learner: logistic |
| 109 | 4 | Accept | 0.29435 | 14.698 | 0.2411 | 0.24441 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 249 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 292 | | 110 | 4 | Accept | 0.37152 | 0.098701 | 0.2411 | 0.24441 | tree | MinLeafSize: 375 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 111 | 4 | Accept | 0.25666 | 5.7386 | 0.2411 | 0.24441 | ensemble | Method: Bag | | | | | | | | | | NumLearningCycles: 61 | | | | | | | | | | LearnRate: NaN | | | | | | | | | | MinLeafSize: 3 | | 112 | 4 | Accept | 0.28059 | 0.10645 | 0.2411 | 0.24441 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 113 | 4 | Accept | 0.28059 | 0.10515 | 0.2411 | 0.24441 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 114 | 6 | Accept | 0.74185 | 2.7447 | 0.2411 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 74.975 |
| 115 | 4 | Accept | 0.78552 | 10.495 | 0.2411 | 0.24441 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.0050713 | | | | | | | | | | Lambda: 3.7406e-06 | | 116 | 4 | Accept | 0.74185 | 2.7544 | 0.2411 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 74.975 | | 117 | 4 | Accept | 0.74185 | 2.6565 | 0.2411 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 74.975 |
| 118 | 4 | Accept | 0.45797 | 11.614 | 0.2411 | 0.24441 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.94716 | | | | | | | | | | KernelScale: 0.072905 |
| 119 | 4 | Accept | 0.65271 | 1.6037 | 0.2411 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010356 | | | | | | | | | | KernelScale: 1.3521 |
| 120 | 4 | Accept | 0.3108 | 1.4953 | 0.2411 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.080708 | | | | | | | | | | KernelScale: 2.7439 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 121 | 4 | Accept | 0.2414 | 1.2954 | 0.2411 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0048396 | | | | | | | | | | KernelScale: 0.02413 |
| 122 | 4 | Accept | 0.5157 | 9.6657 | 0.2411 | 0.24441 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.071659 | | | | | | | | | | Lambda: 1.3447e-05 |
| 123 | 4 | Accept | 0.53186 | 390.32 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 416.46 | | | | | | | | | | KernelScale: 0.0019087 | | 124 | 4 | Best | 0.2402 | 1.4571 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.033468 | | | | | | | | | | KernelScale: 0.073489 |
| 125 | 4 | Accept | 0.2402 | 1.3713 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0040249 | | | | | | | | | | KernelScale: 0.030373 |
| 126 | 3 | Accept | 0.24529 | 39.503 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.32116 | | | | | | | | | | KernelScale: 0.0076281 | | 127 | 3 | Accept | 0.25576 | 1.4031 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.018022 | | | | | | | | | | KernelScale: 0.3703 |
| 128 | 6 | Accept | 0.2426 | 1.3555 | 0.2402 | 0.24229 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.029999 | | | | | | | | | | KernelScale: 0.046245 |
| 129 | 3 | Accept | 0.24499 | 9.5236 | 0.2402 | 0.24419 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.38346 | | | | | | | | | | KernelScale: 0.01786 | | 130 | 3 | Accept | 0.28059 | 0.12828 | 0.2402 | 0.24419 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 131 | 3 | Accept | 0.6886 | 1.4213 | 0.2402 | 0.24419 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.0763 | | | | | | | | | | Learner: logistic | | 132 | 3 | Accept | 0.4487 | 3.111 | 0.2402 | 0.24419 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 4.845 | | | | | | | | | | KernelScale: 0.74028 |
| 133 | 6 | Accept | 0.24469 | 8.3042 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45335 | | | | | | | | | | KernelScale: 0.02034 |
| 134 | 3 | Accept | 0.26204 | 1.7109 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0011217 | | | | | | | | | | KernelScale: 0.10651 | | 135 | 3 | Accept | 0.74185 | 4.0388 | 0.2402 | 0.24441 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 10.518 | | | | | | | | | | Lambda: 0.20458 | | 136 | 3 | Accept | 0.29734 | 9.9598 | 0.2402 | 0.24441 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 107 | | | | | | | | | | LearnRate: 0.0023672 | | | | | | | | | | MinLeafSize: 197 | | 137 | 3 | Accept | 0.44391 | 2.0399 | 0.2402 | 0.24441 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.00025608 |
| 138 | 6 | Accept | 0.24559 | 3.2102 | 0.2402 | 0.24276 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.5211 | | | | | | | | | | KernelScale: 0.04163 |
| 139 | 4 | Accept | 0.25067 | 1.4995 | 0.2402 | 0.24276 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.030306 | | | | | | | | | | KernelScale: 0.37391 | | 140 | 4 | Accept | 0.24559 | 1.6227 | 0.2402 | 0.24276 | linear | Coding: onevsone | | | | | | | | | | Lambda: 3.0973e-07 | | | | | | | | | | Learner: svm | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 141 | 4 | Accept | 0.74634 | 7.0945 | 0.2402 | 0.24276 | kernel | Coding: onevsone | | | | | | | | | | KernelScale: 0.01094 | | | | | | | | | | Lambda: 0.0013866 |
| 142 | 4 | Accept | 0.29076 | 1.3294 | 0.2402 | 0.24334 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.006875 | | | | | | | | | | KernelScale: 0.38629 |
| 143 | 4 | Accept | 0.24379 | 1.4595 | 0.2402 | 0.24269 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010315 | | | | | | | | | | KernelScale: 0.0079737 |
| 144 | 4 | Accept | 0.24499 | 4.9134 | 0.2402 | 0.24226 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.62692 | | | | | | | | | | KernelScale: 0.033311 |
| 145 | 4 | Accept | 0.24469 | 12.208 | 0.2402 | 0.24242 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010122 | | | | | | | | | | KernelScale: 0.0010167 |
| 146 | 4 | Accept | 0.38588 | 1.5037 | 0.2402 | 0.24388 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0011127 | | | | | | | | | | KernelScale: 0.52838 |
| 147 | 4 | Accept | 0.24589 | 1.3026 | 0.2402 | 0.24228 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.1424 | | | | | | | | | | KernelScale: 0.58035 |
| 148 | 4 | Accept | 0.2408 | 1.2582 | 0.2402 | 0.24165 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010168 | | | | | | | | | | KernelScale: 0.032668 |
| 149 | 4 | Accept | 0.24469 | 1.2764 | 0.2402 | 0.24197 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.13008 | | | | | | | | | | KernelScale: 0.53455 |
| 150 | 5 | Accept | 0.24469 | 6.7111 | 0.2402 | 0.24222 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45516 | | | | | | | | | | KernelScale: 0.024796 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 151 | 5 | Accept | 0.30422 | 1.5471 | 0.2402 | 0.24244 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.36805 | | | | | | | | | | KernelScale: 4.7124 |
| 152 | 6 | Accept | 0.24559 | 1.5701 | 0.2402 | 0.2414 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.21039 | | | | | | | | | | KernelScale: 0.63504 |
| 153 | 5 | Accept | 0.24529 | 66.311 | 0.2402 | 0.24243 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.45714 | | | | | | | | | | KernelScale: 0.0069546 | | 154 | 5 | Accept | 0.24589 | 1.604 | 0.2402 | 0.24243 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010712 | | | | | | | | | | KernelScale: 0.043866 |
| 155 | 5 | Accept | 0.29345 | 1.4823 | 0.2402 | 0.24262 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.032053 | | | | | | | | | | KernelScale: 0.88175 |
| 156 | 6 | Accept | 0.25247 | 1.4792 | 0.2402 | 0.24289 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.14103 | | | | | | | | | | KernelScale: 0.91489 |
| 157 | 6 | Accept | 0.2405 | 1.5732 | 0.2402 | 0.242 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010011 | | | | | | | | | | KernelScale: 0.012195 |
| 158 | 6 | Accept | 0.2426 | 1.5232 | 0.2402 | 0.24223 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.52301 | | | | | | | | | | KernelScale: 0.69511 |
| 159 | 6 | Accept | 0.25456 | 1.9467 | 0.2402 | 0.24253 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.63305 | | | | | | | | | | KernelScale: 2.1073 |
| 160 | 5 | Accept | 0.24559 | 133.37 | 0.2402 | 0.24253 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.73914 | | | | | | | | | | KernelScale: 0.005832 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 161 | 5 | Accept | 0.57314 | 8.1723 | 0.2402 | 0.24253 | kernel | Coding: onevsall | | | | | | | | | | KernelScale: 0.071267 | | | | | | | | | | Lambda: 1.4009e-06 |
| 162 | 5 | Accept | 0.24649 | 1.6878 | 0.2402 | 0.24325 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.19825 | | | | | | | | | | KernelScale: 0.74742 |
| 163 | 5 | Accept | 0.2417 | 1.5824 | 0.2402 | 0.24297 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010619 | | | | | | | | | | KernelScale: 0.01008 |
| 164 | 4 | Accept | 0.45528 | 140.46 | 0.2402 | 0.24441 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.00706 | | | | | | | | | | KernelScale: 0.030909 | | 165 | 4 | Accept | 0.31409 | 1.5573 | 0.2402 | 0.24441 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.56217 | | | | | | | | | | KernelScale: 7.5961 |
| 166 | 4 | Accept | 0.46575 | 2.2317 | 0.2402 | 0.24441 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.10607 | | | | | | | | | | KernelScale: 0.67446 |
| 167 | 4 | Accept | 0.24529 | 3.3215 | 0.2402 | 0.24334 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.4843 | | | | | | | | | | KernelScale: 0.069679 |
| 168 | 4 | Accept | 0.25396 | 1.3297 | 0.2402 | 0.24353 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.52671 | | | | | | | | | | KernelScale: 1.9394 |
| 169 | 5 | Accept | 0.2402 | 1.5257 | 0.2402 | 0.24313 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010378 | | | | | | | | | | KernelScale: 0.012749 |
| 170 | 6 | Accept | 0.3464 | 1.8583 | 0.2402 | 0.24237 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.50591 | | | | | | | | | | KernelScale: 9.2835 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 171 | 6 | Accept | 0.24499 | 12.298 | 0.2402 | 0.24203 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0018442 | | | | | | | | | | KernelScale: 0.0012506 |
| 172 | 6 | Accept | 0.30272 | 1.6031 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.8265 | | | | | | | | | | KernelScale: 11.897 |
| 173 | 6 | Accept | 0.24349 | 1.4247 | 0.2402 | 0.24134 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010459 | | | | | | | | | | KernelScale: 0.036404 |
| 174 | 6 | Accept | 0.29794 | 1.6502 | 0.2402 | 0.24299 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 5.4893 | | | | | | | | | | KernelScale: 13.359 |
| 175 | 5 | Accept | 0.24529 | 109.43 | 0.2402 | 0.24345 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.88969 | | | | | | | | | | KernelScale: 0.0075436 | | 176 | 5 | Accept | 0.25007 | 1.6406 | 0.2402 | 0.24345 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010212 | | | | | | | | | | KernelScale: 0.071763 |
| 177 | 5 | Accept | 0.24619 | 2.171 | 0.2402 | 0.24259 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010341 | | | | | | | | | | KernelScale: 0.0039146 |
| 178 | 5 | Accept | 0.32576 | 1.6841 | 0.2402 | 0.24185 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.6102 | | | | | | | | | | KernelScale: 14.532 |
| 179 | 5 | Accept | 0.8262 | 890.49 | 0.2402 | 0.24179 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 0.18649 | | | | | | | | | | KernelScale: 0.0010802 |
| 180 | 5 | Accept | 0.30212 | 1.5671 | 0.2402 | 0.24174 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 6.0833 | | | | | | | | | | KernelScale: 15.913 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 181 | 5 | Accept | 0.24499 | 14.697 | 0.2402 | 0.24192 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0078062 | | | | | | | | | | KernelScale: 0.0020664 |
| 182 | 5 | Accept | 0.24529 | 61.944 | 0.2402 | 0.2414 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.0466 | | | | | | | | | | KernelScale: 0.011265 |
| 183 | 5 | Accept | 0.24499 | 17.644 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0021279 | | | | | | | | | | KernelScale: 0.0010743 |
| 184 | 5 | Accept | 0.29794 | 1.6007 | 0.2402 | 0.24189 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010132 | | | | | | | | | | KernelScale: 0.17669 |
| 185 | 4 | Accept | 0.24918 | 279.94 | 0.2402 | 0.24236 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 891.39 | | | | | | | | | | KernelScale: 0.076868 | | 186 | 4 | Accept | 0.26593 | 1.4006 | 0.2402 | 0.24236 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.14381 | | | | | | | | | | KernelScale: 1.2859 |
| 187 | 4 | Accept | 0.24768 | 1.438 | 0.2402 | 0.24161 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.47763 | | | | | | | | | | KernelScale: 1.4081 |
| 188 | 4 | Accept | 0.24589 | 2.5142 | 0.2402 | 0.2416 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010362 | | | | | | | | | | KernelScale: 0.0024579 |
| 189 | 4 | Accept | 0.2405 | 1.2662 | 0.2402 | 0.24215 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 11.232 | | | | | | | | | | KernelScale: 1.4768 |
| 190 | 4 | Accept | 0.24678 | 1.2604 | 0.2402 | 0.24174 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.48152 | | | | | | | | | | KernelScale: 1.1771 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 191 | 4 | Accept | 0.25426 | 1.2846 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.47194 | | | | | | | | | | KernelScale: 1.7714 |
| 192 | 5 | Accept | 0.24768 | 1.3295 | 0.2402 | 0.24188 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.44397 | | | | | | | | | | KernelScale: 1.3493 |
| 193 | 6 | Accept | 0.2414 | 1.3979 | 0.2402 | 0.24154 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.8796 | | | | | | | | | | KernelScale: 1.4211 |
| 194 | 6 | Accept | 0.24499 | 18.984 | 0.2402 | 0.24225 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0018335 | | | | | | | | | | KernelScale: 0.0010104 |
| 195 | 6 | Accept | 0.25037 | 1.5112 | 0.2402 | 0.24212 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.41413 | | | | | | | | | | KernelScale: 1.4211 |
| 196 | 6 | Accept | 0.24499 | 6.007 | 0.2402 | 0.24191 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010001 | | | | | | | | | | KernelScale: 0.0013198 |
| 197 | 6 | Accept | 0.24529 | 111.16 | 0.2402 | 0.24215 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.046722 | | | | | | | | | | KernelScale: 0.0016388 |
| 198 | 5 | Accept | 0.24499 | 18.688 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0033281 | | | | | | | | | | KernelScale: 0.0012057 | | 199 | 5 | Accept | 0.2426 | 1.4789 | 0.2402 | 0.24186 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.0066 | | | | | | | | | | KernelScale: 1.3889 |
| 200 | 6 | Accept | 0.2417 | 1.499 | 0.2402 | 0.24212 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.7327 | | | | | | | | | | KernelScale: 1.3637 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 201 | 5 | Accept | 0.26473 | 277.53 | 0.2402 | 0.24176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.19781 | | | | | | | | | | KernelScale: 0.0010905 | | 202 | 5 | Accept | 0.30332 | 1.5632 | 0.2402 | 0.24176 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.075228 | | | | | | | | | | KernelScale: 1.8677 |
| 203 | 5 | Accept | 0.3108 | 1.5519 | 0.2402 | 0.24188 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.8086 | | | | | | | | | | KernelScale: 16.078 |
| 204 | 5 | Accept | 0.2414 | 1.4413 | 0.2402 | 0.24147 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.1547 | | | | | | | | | | KernelScale: 0.57043 |
| 205 | 5 | Accept | 0.3108 | 1.5576 | 0.2402 | 0.24195 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.0235 | | | | | | | | | | KernelScale: 16.787 |
| 206 | 5 | Accept | 0.2411 | 1.3939 | 0.2402 | 0.24163 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010447 | | | | | | | | | | KernelScale: 0.022855 |
| 207 | 5 | Accept | 0.29704 | 1.5568 | 0.2402 | 0.24155 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.11574 | | | | | | | | | | KernelScale: 1.8573 |
| 208 | 5 | Accept | 0.2426 | 1.4114 | 0.2402 | 0.24195 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.2749 | | | | | | | | | | KernelScale: 1.2395 |
| 209 | 5 | Accept | 0.24559 | 3.445 | 0.2402 | 0.2417 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 560.48 | | | | | | | | | | KernelScale: 1.4181 |
| 210 | 4 | Accept | 0.24678 | 266.64 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 11.849 | | | | | | | | | | KernelScale: 0.0095181 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 211 | 4 | Accept | 0.24589 | 2.1278 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 41.82 | | | | | | | | | | KernelScale: 0.71147 |
| 212 | 3 | Accept | 0.95034 | 1042.9 | 0.2402 | 0.24153 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 10.672 | | | | | | | | | | KernelScale: 0.0013453 | | 213 | 3 | Accept | 0.24649 | 2.0006 | 0.2402 | 0.24153 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 210.22 | | | | | | | | | | KernelScale: 1.4654 |
| 214 | 6 | Accept | 0.26653 | 1.2521 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.16417 | | | | | | | | | | KernelScale: 1.3626 |
| 215 | 4 | Accept | 0.2429 | 1.5969 | 0.2402 | 0.24136 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010295 | | | | | | | | | | KernelScale: 0.008743 | | 216 | 4 | Accept | 0.33353 | 1.2604 | 0.2402 | 0.24136 | nb | DistributionNames: kernel | | | | | | | | | | Width: 0.0017992 | | 217 | 4 | Accept | 0.28059 | 0.2661 | 0.2402 | 0.24136 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN |
| 218 | 4 | Accept | 0.24798 | 1.3024 | 0.2402 | 0.24173 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 66.132 | | | | | | | | | | KernelScale: 16.829 |
| 219 | 4 | Accept | 0.24529 | 1.255 | 0.2402 | 0.24156 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 1.0858 | | | | | | | | | | KernelScale: 1.3599 |
| 220 | 4 | Accept | 0.2408 | 1.2539 | 0.2402 | 0.24183 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 3.8001 | | | | | | | | | | KernelScale: 1.3322 |
|===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 221 | 4 | Accept | 0.24589 | 1.8906 | 0.2402 | 0.24185 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 399.07 | | | | | | | | | | KernelScale: 2.1791 |
| 222 | 4 | Accept | 0.2414 | 1.2821 | 0.2402 | 0.24187 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 4.8589 | | | | | | | | | | KernelScale: 1.3542 |
| 223 | 4 | Accept | 0.2423 | 1.2804 | 0.2402 | 0.24131 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 2.6082 | | | | | | | | | | KernelScale: 1.3565 |
| 224 | 4 | Accept | 0.24559 | 3.1158 | 0.2402 | 0.24116 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 536.1 | | | | | | | | | | KernelScale: 1.4679 |
| 225 | 4 | Accept | 0.25067 | 1.3228 | 0.2402 | 0.24178 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 74.127 | | | | | | | | | | KernelScale: 18.528 |
| 226 | 3 | Accept | 0.28358 | 328 | 0.2402 | 0.24207 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.47384 | | | | | | | | | | KernelScale: 0.001006 | | 227 | 3 | Accept | 0.24499 | 6.2167 | 0.2402 | 0.24207 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0026293 | | | | | | | | | | KernelScale: 0.0019141 |
| 228 | 6 | Accept | 0.2402 | 1.2501 | 0.2402 | 0.24173 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 0.0010356 | | | | | | | | | | KernelScale: 0.012867 |
| 229 | 3 | Accept | 0.24589 | 1.4248 | 0.2402 | 0.24173 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 104.21 | | | | | | | | | | KernelScale: 17.45 | | 230 | 3 | Accept | 0.43015 | 0.13765 | 0.2402 | 0.24173 | discr | Delta: 0.00951 | | | | | | | | | | Gamma: 0.68613 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 231 | 3 | Accept | 0.47383 | 2.5354 | 0.2402 | 0.24173 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 1.2431 | | | | | | | | | | KernelScale: 0.76632 | | 232 | 3 | Accept | 0.67664 | 0.30377 | 0.2402 | 0.24173 | knn | NumNeighbors: 8 | | | | | | | | | | Distance: jaccard |
| 233 | 6 | Accept | 0.2414 | 1.2369 | 0.2402 | 0.24133 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 820.98 | | | | | | | | | | KernelScale: 17.331 |
| 234 | 3 | Accept | 0.27759 | 1.4452 | 0.2402 | 0.24133 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 22.198 | | | | | | | | | | KernelScale: 17.97 | | 235 | 3 | Accept | 0.28059 | 0.13623 | 0.2402 | 0.24133 | nb | DistributionNames: normal | | | | | | | | | | Width: NaN | | 236 | 3 | Accept | 0.52408 | 1.503 | 0.2402 | 0.24133 | knn | NumNeighbors: 351 | | | | | | | | | | Distance: mahalanobis | | 237 | 3 | Accept | 0.27789 | 5.8867 | 0.2402 | 0.24133 | ensemble | Method: RUSBoost | | | | | | | | | | NumLearningCycles: 68 | | | | | | | | | | LearnRate: 0.18331 | | | | | | | | | | MinLeafSize: 3 |
| 238 | 6 | Accept | 0.24499 | 1.2819 | 0.2402 | 0.24188 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 201.86 | | | | | | | | | | KernelScale: 18.062 |
| 239 | 3 | Accept | 0.25097 | 1.4345 | 0.2402 | 0.24116 | svm | Coding: onevsone | | | | | | | | | | BoxConstraint: 105.8 | | | | | | | | | | KernelScale: 23.315 | | 240 | 3 | Accept | 0.30212 | 0.12275 | 0.2402 | 0.24116 | tree | MinLeafSize: 122 | |===========================================================================================================================================| | Iter | Active | Eval | Validation | Time for training | Observed min | Estimated min | Learner | Hyperparameter: Value | | | workers | result | loss | & validation (sec)| validation loss | validation loss | | | |===========================================================================================================================================| | 241 | 3 | Accept | 0.25307 | 11.978 | 0.2402 | 0.24116 | ensemble | Method: AdaBoostM2 | | | | | | | | | | NumLearningCycles: 119 | | | | | | | | | | LearnRate: 0.60308 | | | | | | | | | | MinLeafSize: 1 | | 242 | 3 | Accept | 0.44182 | 11.435 | 0.2402 | 0.24116 | svm | Coding: onevsall | | | | | | | | | | BoxConstraint: 6.2783 | | | | | | | | | | KernelScale: 0.19364 |
__________________________________________________________ Optimization completed. Total iterations: 242 Total elapsed time: 1907.7403 seconds Total time for training and validation: 4936.5339 seconds Best observed learner is a multiclass svm model with: Coding (ECOC): onevsone BoxConstraint: 0.033468 KernelScale: 0.073489 Observed validation loss: 0.2402 Time for training and validation: 1.4571 seconds Best estimated learner (returned model) is a multiclass svm model with: Coding (ECOC): onevsone BoxConstraint: 0.0010378 KernelScale: 0.012749 Estimated validation loss: 0.24116 Estimated time for training and validation: 1.5595 seconds Documentation for fitcauto display
The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data (creditTrain), the listed Learner (or model) type, and the displayed hyperparameter values.
Evaluate Test Set Performance
The model Mdl corresponds to the best point in the Bayesian optimization according to the 'min-visited-mean' criterion. To gauge how the model will perform on new data, look at the observed cross-validation accuracy of the model (cvAccuracy) and its general estimated performance based on the Bayesian optimization (estimatedAccuracy).
[x,~,iteration] = bestPoint(Results,'Criterion','min-visited-mean'); cvError = Results.ObjectiveTrace(iteration); cvAccuracy = 1 - cvError
cvAccuracy = 0.7598
estimatedError = predictObjective(Results,x); estimatedAccuracy = 1 - estimatedError
estimatedAccuracy = 0.7588
Evaluate the performance of the model on the test set. Create a confusion matrix from the results, and specify the order of the classes in the confusion matrix.
testAccuracy = 1 - loss(Mdl,creditTest,'Rating')testAccuracy = 0.7438
cm = confusionchart(creditTest.Rating,predict(Mdl,creditTest));
sortClasses(cm,{'AAA','AA','A','BBB','BB','B','CCC'})
Input Arguments
Output Arguments
More About
Tips
Depending on the size of your data and the number of learners you specify,
fitcautocan take some time to run. If you have a Parallel Computing Toolbox license, you can speed up computations by running the optimization in parallel. To do so, specify'HyperparameterOptimizationOptions',struct('UseParallel',true). You can include other fields in the structure to control other aspects of the optimization. SeeHyperparameterOptimizationOptions.
Algorithms
Alternative Functionality
If you are unsure which models work best for your data set, you can alternatively use the Classification Learner app. Using the app, you can perform hyperparameter tuning for different models, and choose the optimized model that performs best. Although you must select a specific model before you can tune the model hyperparameters, Classification Learner provides greater flexibility for selecting optimizable hyperparameters and setting hyperparameter values. However, you cannot optimize in parallel, choose
'linear'or'kernel'learners, specify observation weights, or specify prior probabilities in the app. For more information, see Hyperparameter Optimization in Classification Learner App.If you know which models might suit your data, you can alternatively use the corresponding model fit functions and specify the
'OptimizeHyperparameters'name-value pair argument to tune hyperparameters. You can compare the results across the models to select the best classifier. For an example of this process, see Moving Towards Automating Model Selection Using Bayesian Optimization.
Extended Capabilities
See Also
fitcdiscr | fitcecoc | fitcensemble | fitckernel | fitcknn | fitclinear | fitcnb | fitcsvm | fitctree
Introduced in R2020a
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
Select web siteYou can also select a web site from the following list:
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
