Understanding the Bias-Variance trade-off using R

I would advise you not to use the dark mode while reading this blog.

The Bias-Variance trade-off is the most basic yet one of the most important (and sometimes misunderstood) concepts in the field of Machine Learning. We have frequently come across phrases like “high bias leads to under-fitting and high variance leads to over-fitting,” or “simpler models have high bias and low variance, whereas more complicated or sophisticated models have low bias and high variance.” But what do bias and variance really mean, and how do they impact a model’s performance and accuracy?

In this article, we will go through the intuitive understanding and mathematical meaning of bias and variance, see the mathematical relation between bias, variance, and performance of a model, and finally, work on a demonstration to see the effects of model complexity on bias and variance using simulation in R.

Before getting started

Let us first discuss the assumptions and notations of this article

Both bias and variance are statistical concepts that have a variety of uses. However, they will be discussed in this article in terms of an estimator that is trying to fit, explain, or estimate an unknown data distribution.

There is a data generator Y = f(X) + ϵ which is generating data (X, Y) (Data can be repeatedly sampled from the generator, resulting in various sample sets)

• f is some fixed, unknown, true data-generating function. (f represents the systematic information that X provides about Y)
• ϵ is a random error term. It is independent of X and 𝔼(ϵ) = 0 and var(ϵ) = σ²
• X is the independent variable and Y is the dependent variable.

We want to estimate f! (using the sample set we got from the generator)

We have training Data:

We have test Observation (not in our training data!):

We are trying to estimate f such that

What is bias and variance in the current setting?

Bias of an estimator is the difference between the expected estimate and the true values in the data.

Intuitively, bias is a measure of how close or far is the estimator to the true data generator it is trying to estimate.

So, considering this, it is only natural to think that an estimator will have high bias if it does not change too much when a different sample set of the data is thrown at it. This will typically be the situation when an estimator does not have the capacity to properly fit the true data generator. So, simpler models have a higher bias as compared to more complex models.

To understand this above concept a little bit better, let’s see what happens when we sample from a 3rd-order data generating function and use a linear estimator to estimate it.

We can see that the estimator does not change too much when a different sample set of data is thrown at it. This happens because the linear estimator does not have the capacity to properly fit the 3rd-order function.

Variance of an estimator is the amount that the estimator will change given different samples of training data.

Considering the above equation, we can say that an estimator has high variance when the estimator changes its estimate a lot when it's trained over multiple samples of the data. Simply putting it this way, the estimator is flexible or complex enough to perfectly fit the training sample and because of that, there is a significant amount of change in the estimate for different samples of the same data.

Let’s use the same data as in the previous part, and try to fit a 7th order polynomial to it.

We can see that the 7th order polynomial is complex enough to perfectly fit the training data generated using a 3rd order polynomial. And because of this, even a small change in training data leads to high variance.

This suggests that an estimator’s bias and variance are complementary to each other. (An estimator with high bias will have low variance and an estimator with high variance will have low bias)

Evaluating the performance of a Learning method

To evaluate the performance of a Statistical Learning method on any given data set, we need some way to measure how well its predictions actually match the observed data. We need to quantify the extent to which the predicted response value for a given observation is close to the true response value for that observation.

This is where the concept of MSE (Mean-Squared Error) comes into the picture. MSE is a commonly-used measure. And it is given by the following formula.

We don’t really care about how well the method works on the training data. We are interested in how well the method performs on the test (previously unseen) data.

We want to choose the method that gives the lowest test error, as opposed to the lowest training error.

We want to estimate the test error i.e,

How can we select a method that minimizes the test MSE?

• Sometimes, we might have access to a set of observations that were not used in training a learning method. We can then simply evaluate the test MSE using the above equation.
• However, when no test observations are available, selecting a method that minimizes the training MSE might seem to be a reasonable approach (because it looks like training MSE and test MSE are closely related). BUT ITS ACTUALLY NOT!!!
• There is no guarantee that the method with the lowest training MSE will also have the lowest test MSE. (Because many statistical learning methods specifically estimate coefficients so as to minimize the training MSE. For such methods, the training MSE can be quite small, but the test MSE is often much larger)

Bias-variance decomposition

Note: This section contains the mathematical derivation of the test MSE into bias and variance components. I highly recommend working through this. But if it’s not something you’d want to do, feel free to skip to the end of this section.

We know that the test error is

This could be re-written as

Let us consider

Simply writing the test error in terms of A, B, C

We know that

and

So,

Also, A is the randomness from the test data and C is the randomness from the training data. So,

And

From this, we get

Taking the terms in the above equation, one at a time, and expanding them

So, finally, we get

We also know that

So,

So, from the above equation we understand that we can break down the test MSE(i.e the error of an estimator on previously unseen data) into two components:

• irreducible error (the variance of the noise in the data)
• reducible error (bias and variance of the estimator)

We only focus on minimizing the reducible error. Both bias and variance are the sources of this reducible error of an estimator.

In the previous section, we discussed that bias and variance of an estimator are complementary to each other (On increasing bias, variance decreases, and vice versa). How does this property come into play while we are trying to minimize the reducible error?

Bias-Variance trade-off

Because the performance of an estimator depends on both the bias and variance and the complementary nature of bias and variance, it is obvious that there is a trade-off between bias and variance.

Any estimator with little flexibility will not be able to explain all of the data points in the sample and will have high bias and low variance and by extension have high error. On the other hand, an estimator that is extremely flexible will explain all the data points too well (Fails to generalize the data on previously unseen samples and eventually fails to generalize the true data) and will have high variance and low bias and by extension have high error.

We want an estimator that balances bias and variance (It would be able to reduce error better than the ones that favor one extreme over the other)

The above graph is a pictorial representation of the bias-variance trade-off.

Demonstration

We understood the theoretical concepts of the Bias-Variance trade-off so far. Let us build on that and try out this simple demonstration.

The results of the demosntration:

Here, we see that the true data-generating function was a fifth-order polynomial. We try three estimators, a cubic polynomial, a 15th order polynomial, and a 5th order polynomial.

From this demonstration, we can verify our theory that the 3rd-order polynomial does not have the capacity to properly fit the true data generator and has high bias (and low variance). On the other end, the 15th-order polynomial fits the data too well and fails to generalize the data, and changes a lot when different samples are thrown at it. Hence, it does not generalize the true data generator and has high variance (and low bias).

Finally, we see that the 5th-order polynomial closely fits the true data-generating function. This is because it balances well between bias and variance, and by extension the test error.