Nonlinear Least Squares Hyperparameter tuning for variable that subsets data

I am working on a nonlinear Least squares (NLS) problem of the following form:

where

are the variables to be optimized and y is the data used to fit the function. can take values in the range only.

Because of the structure of the problem (let's assume this doesn't accept alternative approaches), I am currently using a gridsearch method to find a "guess" of to use as an input in the NLS problem in order to find . Here, we can think of as some sort of "starting point", which is used to subset data (i.e. pick relevant data) through a boolean variable and then pass that subsample to the NLS problem. In matlab, I am doing something like:

t = linspace(0,1,100);
idx = t > x2;
NLS_objfun = @(theta)(y(idx) - f(t(idx),theta))'(y(idx) - f(t(idx),theta));

Then I minimize NLS_objfun using an optimizer.

The method works and delivers the correct values of with low error. However, it takes a long time.

Thus, I was trying to use bayesopt to tune the parameter and then proceed with the NLS problem (which is really fast conditional on a guess of ). When I am trying to implement this, I do something similar to what follows:

x2 = optimizevariable('x2',[0,1],'Type','real')
fun = @(theta)NLS_objfun(theta);    %for tau_l
results = bayesopt(fun,[x2],'Verbose',0,...
    'AcquisitionFunctionName','expected-improvement-plus')

I get the following error, which of course comes from the subsetting.

Operator '>' is not supported for operands of type 'optimizableVariable'.

I was reading the bayesopt documentation, but I couldn't find more details on how to overcome this structure.

Any help is much appreciated

Thanks!

Answer 1

[Balaji]

Hi Alfonso

I understand that you are facing an error in using a variable of type ‘optimizableVariable’ while trying to perform non-linear optimization.

To access the value of the optimized variable in the variable in an objective function as follows :

x2 = optimizableVariable('x2',[0,1],'Type','real') 
fun = @(theta, x)NLS_objfun(theta, x.x2);    %for tau_l 
results = bayesopt(fun,[x2],'Verbose',0,... 
    'AcquisitionFunctionName','expected-improvement-plus') 

And put the logical condition of the index inside the objective function ‘fun’.

For more information on ‘optimizableVariable’ I suggest you refer to the following documentation:

https://www.mathworks.com/help/stats/optimizablevariable.html

Hope this helps

Thanks

Balaji