Definition
Underfitting in AI occurs when a model is too simplistic or lacks the complexity needed to learn the underlying patterns in the data. This leads to poor performance on both the training and test sets, as the model fails to capture important features of the data. It often happens when the model has insufficient capacity, is overly regularised, or has not been trained for enough epochs.
What is underfitting?
Underfitting happens when a model fails to identify the dominant trends within a dataset. It cannot accurately represent the structure of the data, which leads to weak predictions and unreliable classifications. Common causes include:
- A model that is too simple
- Very high regularisation
- Weak or missing features
- Not enough training time
- Insufficient input data
In practical terms, an underfitted model is like drawing a straight line through curved data. The line misses the underlying pattern, so predictions based on it are flawed from the start. Because underfitting is visible even on the training dataset, it is often easier to detect than overfitting.
High bias and low variance
Underfitting is strongly linked to the bias variance trade off. In this case, the model exhibits:
- High bias
- Low variance
High bias means the model makes strong assumptions and ignores important patterns. It oversimplifies the problem. For example, assuming all birds are small and can fly would ignore large birds such as ostriches or penguins. The model becomes biased in its predictions.
Low variance means the model produces similar outputs even when the data changes. It is not sensitive enough to variations in the dataset.
In short: Underfitting equals high bias plus low variance.
Underfitting versus overfitting
Underfitting is the opposite of overfitting.
- Underfitting: The model is too simple and performs poorly on both training and test data
- Overfitting: The model is too complex and performs well on training data but poorly on test data
Overfitting often occurs when a model memorises training data, including noise and outliers. This leads to low bias but high variance.
Identifying overfitting can be more challenging because training accuracy appears high. Techniques such as k fold cross validation are used to assess how well a model generalises. In k fold cross validation, data is split into equally sized subsets. Each subset takes a turn as the validation set while the others are used for training. The results are averaged to evaluate overall performance.
The goal in model development is to find the balance between underfitting and overfitting. This balanced point allows the model to generalise effectively to new data.
Why underfitting is a problem
A model that cannot generalise cannot be trusted for real world use. Generalisation is what allows machine learning systems to:
- Classify new inputs
- Make accurate predictions
- Adapt to unseen scenarios
If a model is underfitted, it misses important effects and may underestimate variability in the data. This can result in misleading conclusions and unreliable outputs.
How to avoid underfitting
Since underfitting is visible in training performance, it can often be addressed early. The aim is to introduce enough complexity for the model to capture meaningful patterns without drifting into overfitting.
Techniques to reduce underfitting include:
- Use a more complex model
- Add new or more relevant features through feature engineering
- Reduce regularisation
- Train for more epochs
- Scale features properly
- Increase the amount of training data
- Try a different algorithm better suited to the problem
- Use ensemble methods to combine multiple models
Regularisation deserves particular attention. While it is commonly used to reduce overfitting by penalising large parameter values, too much regularisation can oversimplify the model. Reducing it carefully can restore the model’s ability to detect dominant trends.
As with most aspects of machine learning, balance is key. Increasing training time or complexity can help, but excessive adjustments risk tipping the model into overfitting.
Key takeaways
- Underfitting occurs when a model is too simple to capture real patterns in the data
- It leads to high error rates on both training and unseen data
- Underfitting is characterised by high bias and low variance
- It is often easier to detect than overfitting because training accuracy is low
- Solutions include increasing model complexity, reducing regularisation and improving feature selection
- The ultimate goal is to strike the right balance between bias and variance for strong generalisation performance