The general workflow for training a SVR model using PSO optimization is as follows:
The parameters we need to find are the hyperparameters of the SVR Model.
This is done by training the SVR model and finding the loss from the predictions made on the validation set using the swarm hyperparameters. I have used Root Mean Squared Error as this is a regression task. These hyperparameters will be updated according to the PSO algorithm. To clarify, we are not training the SVR using PSO (we do not need to consider the objective function of the SVR). We are using PSO to find optimal hyperparameters for the SVR Model.
This can be done as follows:
load carsmall
% Use Horsepower and Weight as features, MPG as the target variable
X = [Horsepower, Weight];
y = MPG;
% Remove any rows with NaN values
validIdx = ~any(isnan(X), 2) & ~isnan(y);
X = X(validIdx, :);
y = y(validIdx);
% Split the data into training and validation sets
cv = cvpartition(length(y), 'HoldOut', 0.3);
X_train = X(training(cv), :);
y_train = y(training(cv));
X_val = X(test(cv), :);
y_val = y(test(cv));
% Define the objective function using RMSE
function rmse = svrObjective(params, X_train, y_train, X_val, y_val)
C = params(1);
epsilon = params(2);
kernelScale = params(3);
% Train SVR model with RBF kernel
svrModel = fitrsvm(X_train, y_train, 'KernelFunction', 'rbf', ...
'BoxConstraint', C, 'Epsilon', epsilon, 'KernelScale', kernelScale);
% Predict on validation set
predictions = predict(svrModel, X_val);
% Calculate root mean squared error
rmse = sqrt(mean((y_val - predictions).^2));
end
% PSO optimization using particleswarm with reasonable bounds
nvars = 3; % Number of variables: C, epsilon, kernelScale
lb = [0.1, 0.001, 0.1]; % Lower bounds for C, epsilon, kernelScale
ub = [10000, 2, 10000]; % Upper bounds for C, epsilon, kernelScale
% Define the objective function handle
objectiveFunction = @(params) svrObjective(params, X_train, y_train, X_val, y_val);
% Set optimization options
options = optimoptions('particleswarm', ...
'SwarmSize', 200, ...
'Display', 'iter');
% Run PSO
rng(42)
[bestParams, bestRMSE] = particleswarm(objectiveFunction, nvars, lb, ub, options);
% Display results
fprintf('Best Parameters: C = %.3f, epsilon = %.3f, kernelScale = %.3f\n', bestParams(1), bestParams(2), bestParams(3));
fprintf('Best RMSE: %.3f\n', bestRMSE);
These parameters can be used to train a SVR Model that minimizes the RMSE.
I have provided a wide range as bounds of the parameters. You may experiment with different bounds, or you may explore other optimization options like SwarmSize, FunctionTolerance, etc.
You can learn more about these options in the documentation page of particleswarm function:
You might also want to look at the documentation page for fitrsvm for other ways to find optimal hyperparameters and have a look at the different hyperparameters:
I hope this helps!
