How to export the overview table from the Experiment Manager App

3 views (last 30 days)
I started using the Experiment Manager App to explore a Neural Network performance on a calssification task and how it depends on the hyperparameters of the model.
The overall design of the App is quite appealing for a first look but the options for exporting the results for further analysis are too limited.
After a hyperparameter sweep the app displays a table of results that displays the accuracy and loss for each trial together with the choosen hyperparameters.
It would be extreemly useful to export this table e.g. to plot validation and test accuracy as function of the hyperparameters to gain more insights. However I could not find such an export option. It is only possible to export the parameters of individual trained Networks. Please let me know if this option exists?
If not, It would be great if such an export option could be added to the app. Or if a function could be provided to extract the results parameters from the results subfolders.
Thanks and Kind Regards
Sebastian Landwehr

Accepted Answer

Anshika Chaurasia
Anshika Chaurasia on 21 Jan 2021
Hi Sebastian,
You can try the following script which will give a table containing validation and training loss/accuracy for all trials:
% sweep intialLearnRate hyperparameter
myInitialLearnRate = [0.0025 0.0050 0.0075];
trialdetail = cell(numel(myInitialLearnRate),1);
for i = 1:3
% params struct containing value of hyperparameter
params = struct('myInitialLearnRate',myInitialLearnRate(i));
% passing hyperparameter value to experiment setup function
[imds,layers,options] = Experiment1_setup1(params);
% train the network
[net,info] = trainNetwork(imds,layers,options);
% storing the Training information such as
% Training loss, Training Accuracy, Validation Loss,
% Validation Acc for a trial
trialdetail{i,:} = info;
% storing information for table
Training_acc(i) = info.TrainingAccuracy(end);
Training_loss(i) = info.TrainingLoss(end);
Validation_acc(i) = info.FinalValidationAccuracy;
Validation_loss(i) = info.FinalValidationLoss;
end
Trial = 1:3;
% Table containing Training informations for all trials
Table = table(Trial(:),Training_acc(:),Training_loss(:),Validation_acc(:),Validation_loss(:),...
'VariableNames',{'Trial','Training Acc','Training Loss','Validation Acc','Validation Loss'});
function [imds,layers,options] = Experiment1_setup1(params)
% This template function constructs an image data store as a training data set,
% a layer graph, and training options. Outputs of this setup function are used
% to call the trainNetwork function.
% Modify this function as a template to set up your own experiment.
digitDatasetPath = fullfile(matlabroot,'toolbox','nnet', ...
'nndemos','nndatasets','DigitDataset');
imds = imageDatastore(digitDatasetPath, ...
'IncludeSubfolders',true, ...
'LabelSource','foldernames');
numTrainingFiles = 750;
[imds,imdsValidation] = splitEachLabel(imds,numTrainingFiles);
inputSize = [28 28 1];
numClasses = 10;
layers = [
imageInputLayer(inputSize)
convolution2dLayer(5,20)
batchNormalizationLayer
reluLayer
fullyConnectedLayer(numClasses)
softmaxLayer
classificationLayer];
options = trainingOptions('sgdm', ...
'MaxEpochs',5, ...
'ValidationData',imdsValidation, ...
'ValidationFrequency',30, ...
'InitialLearnRate',params.myInitialLearnRate);
end
Refer to this documentation for more information on Experiment1_setup1 function.
I have brought your idea of export option to the notice of our developers. They may investigate the matter further.
  4 Comments
Payman Goodarzi
Payman Goodarzi on 30 Nov 2021
Although is no export function for an experiment table, it is possible to extract its variables from the Results folder, the structure is understandable.
My problem is that I can not find the "custom metric", where is it stored?
Vishnu Keyen
Vishnu Keyen on 28 Jan 2022
I am sorry for not posting the code with it.
The custom metric is specified as a function in the experiment manager.

Sign in to comment.

More Answers (0)

Products


Release

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!