(Feature Request)Is there a good way to use the Optuna hyperparameter optimization framework in MATLAB?

As we all know, Optuna is a well-regarded hyperparameter optimization framework that is independent of any machine learning framework and is very easy to use in Python. However, my cost object-function is in MATLAB, which typically belongs to black-box optimization, and I am unsure how to use the Optuna library.

I also know that the Statistics and Machine Learning Toolbox has techniques like random search, grid search, and Bayesian hyperparameter optimization, but they haven't performed very well. The Global Optimization Toolbox can use heuristic search algorithms such as GA and PSO, but I am limited by the high computational cost of the cost function. I also tried surrogateopt. Although the iteration speed is relatively fast, it tends to get stuck in local optima, causing subsequent iterations to nearly stop. Overall, it doesn't perform as well as the PSO algorithm! Therefore, I would like to explore the performance of the Optuna library.

-----------------2025.12.11 update------------------------

Simple Example:

fun = @(x)(x-2).^2;
x0=0;
A =[];
b = [];
Aeq = [];
beq = [];
lb = -10;
ub = 10;
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub)

Local minimum found that satisfies the constraints.

Optimization completed because the objective function is non-decreasing in 
feasible directions, to within the value of the optimality tolerance,
and constraints are satisfied to within the value of the constraint tolerance.

x = 2.0000

install python optuna library

websave("get-pip.py","https://bootstrap.pypa.io/get-pip.py");
!python get-pip.py

Defaulting to user installation because normal site-packages is not writeable
Collecting pip
  Using cached pip-25.3-py3-none-any.whl.metadata (4.7 kB)
Using cached pip-25.3-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.3
    Uninstalling pip-25.3:
      Successfully uninstalled pip-25.3
□[33m  WARNING: The scripts pip, pip3 and pip3.10 are installed in '/home/matlab/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.□[0m□[33m
□[0mSuccessfully installed pip-25.3

!python -m pip --version

pip 25.3 from /home/matlab/.local/lib/python3.10/site-packages/pip (python 3.10)

!python -m pip install numpy optuna

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: numpy in /home/matlab/.local/lib/python3.10/site-packages (2.2.6)
Requirement already satisfied: optuna in /home/matlab/.local/lib/python3.10/site-packages (4.6.0)
Requirement already satisfied: alembic>=1.5.0 in /home/matlab/.local/lib/python3.10/site-packages (from optuna) (1.17.2)
Requirement already satisfied: colorlog in /home/matlab/.local/lib/python3.10/site-packages (from optuna) (6.10.1)
Requirement already satisfied: packaging>=20.0 in /home/matlab/.local/lib/python3.10/site-packages (from optuna) (25.0)
Requirement already satisfied: sqlalchemy>=1.4.2 in /home/matlab/.local/lib/python3.10/site-packages (from optuna) (2.0.45)
Requirement already satisfied: tqdm in /home/matlab/.local/lib/python3.10/site-packages (from optuna) (4.67.1)
Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from optuna) (6.0.1)
Requirement already satisfied: Mako in /home/matlab/.local/lib/python3.10/site-packages (from alembic>=1.5.0->optuna) (1.3.10)
Requirement already satisfied: typing-extensions>=4.12 in /home/matlab/.local/lib/python3.10/site-packages (from alembic>=1.5.0->optuna) (4.15.0)
Requirement already satisfied: tomli in /home/matlab/.local/lib/python3.10/site-packages (from alembic>=1.5.0->optuna) (2.3.0)
Requirement already satisfied: greenlet>=1 in /home/matlab/.local/lib/python3.10/site-packages (from sqlalchemy>=1.4.2->optuna) (3.3.0)
Requirement already satisfied: MarkupSafe>=0.9.2 in /home/matlab/.local/lib/python3.10/site-packages (from Mako->alembic>=1.5.0->optuna) (3.0.3)

Below is the my call optuna for optimization. How can I best pass the MATLAB cost function fun and its constraints to optuna? I want call optuna from MATLAB.

study = py.optuna.create_study()

[I 2025-12-11 07:34:52,821] A new study created in memory with name: no-name-bec3ab28-bca9-4d4c-94d9-abc7e7e51676

study = 
  Python Study with properties:

     best_trials: [1×0 py.list]
       direction: [1×1 py.optuna.study._study_direction.StudyDirection]
      directions: [1×1 py.list]
    metric_names: [1×1 py.NoneType]
    system_attrs: [1×1 py.dict]
          trials: [1×0 py.list]
      user_attrs: [1×1 py.dict]
      study_name: [1×44 py.str]
         sampler: [1×1 py.optuna.samplers._tpe.sampler.TPESampler]
          pruner: [1×1 py.optuna.pruners._median.MedianPruner]

    <optuna.study.study.Study object at 0x76ec4ac4a260>

study.optimize(fun, n_trials=100)

Error using py.optuna.study.study.Study/optimize
Handle to MATLAB function '@(x)(x-2).^2' is not supported. Use a handle to a Python function.

original native python example:

import optuna

def objective(trial):
    x = trial.suggest_float('x', -10, 10)
    return (x - 2) ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=100)

study.best_params  # E.g. {'x': 2.002108042}

Answer 1

[xingxingcui]

After exploration and investigation, it is regrettable that MATLAB currently cannot effectively pass the handle of the optimization objective function to the Optuna framework, especially for complex objective functions that may involve passing extra parameter data. However, for passing ordinary general data, most built-in types can be smoothly exchanged between MATLAB and Python.

Call Python from MATLAB

I believe that in the future, MATHWORKS will enhance considerations in this area!

• Statistics and Machine Learning Toolbox™
• Global Optimization Toolbox™

[Andreas Goser]

Thank you for feeding your finding back to the community. I have looked into two MathWorks-internal databases and cannot find knowledgebase articles or enhancement requests in that space. But maybe I am searching incorrectly as I am not a domain expert.

When you write "I believe that in the future" - have you discussed your need with a MathWorks person and this person will create and enhancement request? Or did you intend to write "I hope"? Then I would recommend contactin Techncal Support so this enhancement request can be logged.

[xingxingcui]

Thank you for your attention to my question. This enhancement requests was not submitted to Technical Support; it was a spontaneous “expectation” on my part that led to your misunderstanding. I apologize for that. Could you help me submit this feature request? As an internal member, you may have an easier time getting support than I do.