Bias and Variance in Machine Learning

When we train machine learning models, we might not always get very good results. We often run into situations where overfitting or underfitting occurs due to which the performance of the machine learning model deteriorates. This happens due to bias and variance errors. In this article, we will discuss these errors in machine learning and how to identify them.

What is Bias in Machine Learning?

When we train a machine learning model on a dataset, it captures the patterns in the data and uses them for predicting results for new data points or test data. The predictions by the machine learning model might not be equal to the expected or actual target values for the test data. The difference between the actual and predicted target values is an error and is termed an error due to bias.

We can define bias as “the inability of a machine learning model to capture the true relationship between the data points and the target values.”

Each ML algorithm works on some assumptions. Hence, each machine learning model has an inherent bias due to the assumptions. For example, a linear regression model expects the data points and target values to be linearly independent. Similarly, the K-Means clustering algorithm assumes that the clusters in the input data are elliptical/circular in shape. On the other hand, the Naive Bayes classification algorithm assumes that the features in the input data are independent of each other.

These assumptions aren’t always valid for real-world data. Due to this, we encounter the following biases.

• Low Bias: A low-bias model makes fewer assumptions about the data or the target function on which we train the model. Examples of low-bias models include decision trees and the K-Nearest Neighbors algorithm. 
• High Bias: High bias comes in a machine learning model if it makes many assumptions and doesn’t actually capture the features of the training data. Due to this, the predictions of a high-bias model are often inaccurate. Examples of machine learning algorithms with high bias are linear regression, Naive Bayes algorithm, logistic regression, etc. These models are easy to train due to bias but they perform well on only those datasets that conform to the assumptions.

What is Variance in Machine Learning?

As the name suggests, variance measures the amount of variation in the predictions of a machine-learning model. It specifies the extent to which a model can adjust depending on the given data set. Mathematically, variance is the measure of how much a predicted variable differs from its expected value. Again, variance can be of two types i.e. low variance and high variance.

• Low variance: Low variance for a machine learning algorithm specifies that there is a very small change in predictions when we change the input dataset. Ideally, a machine learning model should not vary the results much when the input data is changed. In such a situation, it is considered that the model has a good understanding of the relationships between different attributes in the dataset.
• High variance: A model with high variance shows a large variation in the predictions when we change the input data. When we train a machine learning algorithm that has a high variance, the model learns a lot and performs well with the training data. However, when we pass an unseen dataset as input, it shows large variations in the output predictions. This isn’t a desired situation.

How to Identify Bias and Variance Errors?

To identify the bias and variance errors, you can use the following tricks.

• If a machine learning model performs well on training data but has a high error rate for test data or any unseen data, it will have high variance.
• If a model shows large errors in seen as well as unseen data, it will have a high bias.

Thus, small training and large test errors denote high variance. Large training and large test errors show high bias for a given machine learning algorithm.

Bias-Variance Tradeoff: What is The Optimal Solution?

Based on the situation, we can have one of the following situations for bias and variance errors in a machine learning model.

• Low variance- low bias: If a machine learning model has low variance and low bias, it will perform best and is an ideal situation for us. However, it is not possible to achieve low variance and low bias for a model in practical situations as real-world data doesn’t conform to any theoretical assumption. Hence, no algorithm is perfect for a given dataset.
• Low variance- high bias: When a model has low variance and high bias, the predictions are consistent. However, they are inaccurate. In this situation, the machine learning algorithm is trained with very few parameters and the model doesn’t learn well from the training data.
• High variance- low bias: When a machine learning model has high variance and low bias, the predictions will be inconsistent. However, they are accurate. In this situation, the machine learning model leads to overfitting and doesn’t generalize well. This is due to the reason that the machine learning algorithm is trained on a large number of parameters.
• High variance- high bias: In case of high variance and high bias, the predictions of a machine learning model are inconsistent as well as inaccurate. This is not a desired situation at all.

When we train a machine learning model, we need to make adjustments between bias and variance to achieve optimum performance. Bias and variance are inversely correlated. Due to this, when we increase bias, variance will decrease. Similarly, when we decrease the bias, variance increases. 

For a model to perform best, bias and variance both should be low. However, we cannot achieve this situation. Hence, we try to achieve the parameter values of a given algorithm for which bias and variance are optimal. In this case, our machine learning model will generalize well. At the same time, it will also have an acceptable bias.

Conclusion

In this article, we discussed bias and variance in machine learning. To learn more about machine learning algorithms, you can read this article on entity embedding in Python. You might also like this article on label encoding. 

I hope you enjoy this reading this article. Stay tuned for more informative articles.

Happy Learning!