Working with LSTM and Bayes Optimization

I am trying to use bayesoptimization to tune the parameters

optimvars = [
    optimizableVariable('InitialLearnRate',[1e-2 1],'Transform','log')
    optimizableVariable('L2Regularization',[1e-10 1e-2],'Transform','log')];

layers = [ ...
    sequenceInputLayer(inputSize,'Normalization','zscore')
    bilstmLayer(numHiddenUnits,'OutputMode','last')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

maxEpochs =25;

options = trainingOptions('adam',...
    'ExecutionEnvironment','cpu',...
    'GradientThreshold',1,...
    'MaxEpochs',maxEpochs,...
    'MiniBatchSize',miniBatchSize, ...
    'SequenceLength', 'longest', ...
    'Shuffle','every-epoch', ...
    'Verbose', 1, ...
    'InitialLearnRate',optimvars.InitialLearnRate,...
    'L2Regularization',optimvars.L2Regularization,...
    'Plots','training-progress');


objFcn = makeObj(Xtrain,YTrain);

bayesObj = bayesopt(objFcn,optimvars, ...
    'MaxTime', 14*60*60, ...
    'IsObjectiveDeterministic',false,...
    'UseParallel',false);

Where am i going wrong as i get the following error:

Unrecognized method, property, or field 'InitialLearnRate' for class 'optimizableVariable'.
Error in AllVsIndx (line 236)
    'InitialLearnRate',optimvars.InitialLearnRate,...

The documentation regarding bayesian optimization is very vague especially when it comes to implementation with LSTM networks

Any help would be appreciated

Thanks

Answer 1

[Jorge Calvo]

If you have R2020b or later, you can use the Experiment Manager app to run Bayesian optimization to determine the best combination of hyperparameters. For more information, see https://www.mathworks.com/help/deeplearning/ug/experiment-using-bayesian-optimization.html.

[CHRISTOPHER MILLAR]

Thanks Jorge

I have just installed R2021a on my machine and will take advantage of this new app

Answer 2

[Don Mathis]

You might find this similar example useful: https://www.mathworks.com/matlabcentral/answers/457788-lstm-time-series-hyperparameter-optimization-using-bayesian-optimization?s_tid=answers_rc1-2_p2_MLT

Answer 3

[Jorge Calvo]

I thought you would like to know that, in R2021b, we are included an example for training long short-term memory (LSTM) networks using Bayesian optimization in Experiment Manager:

https://www.mathworks.com/help/deeplearning/ug/exp-mgr-sequence-regression-example.html

I hope you find it helpful!

[CHRISTOPHER MILLAR]

Hi jorge,

Yes I have been using it and it makes the optimization process very easy to use.

Thanks for updating the thread with this information.

Regards

Chris