Definition
A parameter set to control the learning process, established by the model designer and not learnt by the model from data. These parameters can directly affect how well a model trains.
What are hyperparameters?
Hyperparameters can be broadly classified into two categories:
- Model hyperparameters define the structure of the model, such as the number of hidden layers in a neural network or the number of neurons per layer.
- Algorithm hyperparameters control the learning process, including the learning rate, batch size, or number of training epochs.
Some algorithms, like ordinary least squares regression, require no hyperparameters, while others, such as LASSO regression, introduce regularisation hyperparameters that must be defined ahead of training.
What is hyperparameter tuning?
Hyperparameter tuning is the process of testing different configurations to find the combination that minimises the model’s loss function and maximises performance. Well-tuned hyperparameters help models:
- Learn efficiently without overfitting or underfitting data.
- Deliver consistent results across different datasets.
- Reduce computational resources during training.
- Improve robustness and reproducibility in research and real-world applications.
Poorly chosen hyperparameters can lead to models that are too simplistic, too complex, or unstable, producing unreliable predictions.
Common hyperparameters
Neural networks
- Learning rate and learning rate decay
- Batch size and momentum
- Number of hidden layers and neurons per layer
- Activation function and dropout rate
- Number of epochs
Support vector machines (SVM)
- C parameter controlling error tolerance
- Kernel type defining data separation
- Gamma controlling influence of support vectors
XGBoost
- Learning rate
- Number of trees (n_estimators)
- Maximum depth of trees
- Minimum child weight
- Subsample ratio
Each hyperparameter affects how the model learns from data, and some interact with others in complex ways, making tuning a careful balancing act.
Methods for hyperparameter tuning
Several strategies exist for finding optimal hyperparameters:
- Grid search: Tests all possible combinations exhaustively but can be computationally expensive.
- Randomised search: Samples hyperparameter values from statistical distributions and is faster than grid search.
- Bayesian optimisation: Uses prior results to probabilistically select promising hyperparameters, improving efficiency.
- Hyperband: Combines random search with early stopping to discard poor configurations quickly.
The choice of tuning method depends on the model complexity, computational resources, and time constraints.
Hyperparameters vs Parameters
Parameters are learned from data during training, while hyperparameters are set before training begins. The values chosen for hyperparameters directly influence the parameters the model learns and, ultimately, the model’s performance. Effective hyperparameter tuning ensures the learning algorithm extracts meaningful patterns from the data while remaining efficient and generalisable.
Key Takeaways
- Hyperparameters are set before training and control how a model learns and performs.
- Model parameters are learned during training and are affected by the chosen hyperparameters.
- Proper hyperparameter tuning maximises accuracy and computational efficiency.
- Poor hyperparameter choices can lead to underfitting, overfitting, or unstable models.
- Different models require different hyperparameters, making tuning essential for optimal performance.