Hyperparameter optimization

In machine learning, hyperparameter optimization^[1] or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process.

Hyperparameter optimization finds a tuple of hyperparameters that yields an optimal model which minimizes a predefined loss function on given independent data.^[2] The objective function takes a tuple of hyperparameters and returns the associated loss.^[2] Cross-validation is often used to estimate this generalization performance, and therefore choose the set of values for hyperparameters that maximize it.^[3]