Getting different results training on the same datasets each time

I'm using 'fitrgp' to train my GPR model. I've specified hyperparameters I'd like it to optimize, but I'm getting slightly different results each time I train it on the same dataset. I know this is a feature of ML in general, but I'd like to know if there's any way to get a reproducible result each time.

I'm already using rng("default") and using the "expected-improvement-plus" acquisition function to improve the reproducibility.

I'd really appreciate any insight into if this is possible!

[the cyclist]

If you have rng("default"), and no intervening calls to rng that affect the seed, I would expect you to get exactly the same results.

Can you post your code?

[Katy]

From https://www.mathworks.com/help/stats/predict-battery-soc-using-machine-learning.html

rng("default")
Mdl = fitrgp(trainData,"SOC", ...
    OptimizeHyperparameters=["BasisFunction","KernelFunction","Standardize"], ...
    HyperparameterOptimizationOptions= ...
    struct(AcquisitionFunctionName="expected-improvement-plus",UseParallel=true))

My code is similar to this as its based off the based off the example here.

One thing I investigated was whether this had to do with the parallel processing. However, even after trying to set the random seeds for the workers, I still am not getting reproducble results either.

https://www.mathworks.com/help/parallel-computing/control-random-number-streams-on-workers.html

[the cyclist]

Sorry I did not see this reply earlier.

This documentation discusses reproducibility in parallel computations. It seems to have some distinct recommendations from the page you linked.

Answer 1

[Ayush Anand]

Hi Katy,

I understand that you are doing Gaussian process regression and getting slightly different results every time, you train the model on the same dataset.

The code that you posted shows that you are using parallel computation option while training. Since MATLAB parallel computing toolbox is inherently non-deterministic in nature, generally it is not sought out when reproducible results are required, and serialization is preferred.

In this case, for producing serialized results, you can set “UseParallel” to false and keep the “rng(“default”)” statement. This disables parallel computations and setting the seed to default every time ensures reproducible results.

The MathWorks Documentation linked below is referring to a workaround to produce reproducible results with parallel computing toolbox, but it doesn’t seem to be possible to inculcate this with the “fitrgp” function:

https://in.mathworks.com/help/stats/reproducibility-in-parallel-statistical-computations.html

Also, as you mentioned in the question, most of the ML algorithms (including Gaussian Process regression) inherently have some randomness due to re-initializations and samplings, so you are bound to get slightly different results.

I hope this helps.