[ad_1]
2.1. Support Vector Machines and Iris Data Set
In a previous post I used Grid Search, Random Search and Bayesian Optimization for hyperparameter optimization using the Iris data set provided by scikit-learn. Iris data set includes 3 different irises petal and sepal lengths and is a commonly-used data set for classification exercises. In this post, we will use the same data set but we will use a Support Vector Machine (SVM) as a model with two parameters that we can optimize as follows:
C: Regularization parameter, which trades off misclassification of training examples against simplicity of the decision surface.gamma: Kernel coefficient, which defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected.
Since the goal of this exercise is to go through the hyperparameter optimization, I will not go deeper into what SVMs do but if you are interested, I find this scikit-learn post helpful.
We will generally follow the same steps that we used in the simple example earlier but will also visualize the process at the end:
1. Import necessary libraries and packages
2. Define the objective function and the search space
3. Run the optimization process
4. Visualize the optimization
2.1.1. Step 1 — Import Libraries and Packages
Let’s import the libraries and packages and then load the data set.
# Import libraries and packages
from sklearn import datasets
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score# Load Iris dataset
iris = datasets.load_iris()
X = iris.data
y = iris.target
2.1.2. Step 2 — Define Objective Function and Search Space
Let’s first start with defining the objective function, which will train an SVM and returns the negative of the cross-validation score — that is what we want to minimize. Note that we are minimizing the negative of cross-validation score to be consistent with the general goal of “minimizing” the objective function (instead of “maximizing” the cross-validation score).
def objective_function(parameters):
clf = SVC(**parameters)
score = cross_val_score(clf, X, y, cv=5).mean()
return -score
Next we will define the search space, which consists of the values that our parameters of C and gamma can take. Note that we will use Hyperopt’s hp.uniform(label, low, high), which returns a value uniformly between “low” and “high” (source).
# Search Space
search_space = {
'C': hp.uniform('C', 0.1, 10),
'gamma': hp.uniform('gamma', 0.01, 1)
}
2.1.3. Run Optimization
Same as the simple example earlier, we will use a TPE algorithm and store the results in a Trials object.
# Trials object to store the results
trials = Trials()# Run optimization
best = fmin(fn=objective_function, space=search_space, algo=tpe.suggest, trials=trials, max_evals=100)
Results:
Source link
