Stochastic Optimization

It finds the minimum of optimization problems by searching randomly.

his algorithm was initially developed back in 1990 during an optimization course given by Andy Hrymak, an excellent professor at McMaster Univeristy.

Most optimization algorithms depend on the initial starting point, thus this algorithm exploits this “weakness” by randomly searching for the minimum.

It needs only the domain of each variable (example: x is between -10 and 10). It will generate random samples (recommended 1000), and after the initial iteration will focus part of the next random samples (recommended 20 percent) to be randomly normally distributed around the best result (minimum) that has been found using the distance of successive best results as the standard deviation.

The algorithm stops based on a minimum change in the x variables (recommended 10-9), or on minimum improvement in the y variable that needs to be minimized (recommended 10-16), or on a maximum number of iterations (recommended 100). All three of these conditions are needed so the algorithm will stop no matter the nature of the optimization problem (highly desirable feature in real-time applications that many times i overlooked).

The user provides the function that needs to be optimized in an m-file. The “fun_optimize.m” provides a variety of functions as examples. Many of the examples are from Simon Fraser University (https://www.sfu.ca/~ssurjano/boha.html). As you will see in the examples provided, the algorithm (although not an efficient one) can handle many interesting problems including constrains.

The basic form of the algorithm is provided here that can handle functions with real numbers. It is written in a form that is easily readable.

In principle any random distribution can be used. Here the uniform and normal distributions are used.

The algorithm can be modified relatively easily to handle also integer problems and mix-integer problems.

The algorithm needs significant modification but it can handle variables that are categorical in nature (x = [ green, blue yellow, red, cyan]).