Nelder-Mead optimization with equality constraints

Hello

I would like to use the function fminsearch of matlab to search for the best hyperparameters of my SVM with a weighted RBF kernel classifier. fminsearch uses the Nelder-Meader simplex method.

Let's say I have the following hyperparameters: C, gamma, w1....wn where wi are the weights of the kernel.

Additionally, I have the constraint that sum(wi) = 1, i.e. all weights must sum up to one.

Is there a possibility to use Nelder-Mead with this equality constraint? I know that there is the fminsearchbnd method for Matlab but I think it can handle only boundary inequality constraints.

Answer 1

[Alan Weiss]

I would imagine that you also have the constraints that several or all of your parameters must be positive.

It might be difficult to have all those constraints satisfied at the same time using just fminsearch, or even fminsearchbnd. You might want to use fmincon or patternsearch to ease your search for the best hyperparameters.

It is likely that you would do best by setting up a rough grid of hyperparameters, search them, and then start a local search using patternsearch or fmincon starting at the best grid point.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation

[Sepp]

Hi Alan

Thank you very much for your answer, I really appreciate it.

Yes, you are right, all parameters must be positive, i.e. 0 <= w(i) <= 1

I have to clarify a bit. I'm using a SVM classifier and the weights are used in a weighted RBF kernel (one weight for each feature). The parameters to estimate are thus C, gamma and the weights. The cost function is the accuracy.

An option would be to allow w(i) to go above 1, i.e. only having the constraint 0 <= w(i). Then I could just normalize the w(i) to sum to one inside my cost function. Would this work with Nelder-Mead or would the method get stuck?

Does fmincon only work for smooth and linear cost function? Because I think my cost function (accuracy) is not linear.

Regarding your suggestion to first using a grid, I think I could also using random search or simulated annealing as a start.