The Perils of Pursuing a Perfect R-Squared

As data scientists and machine learning practitioners, we often find ourselves searching for models that fit our data as closely as possible. One of the most common metrics used to evaluate the fit of a linear regression model is the coefficient of determination, better known as R-squared ($R^2$).

What is R-Squared?

R-squared measures the proportion of variance in the dependent variable that can be explained by the independent variable(s) in a regression model. It provides an indication of goodness of fit and predictive power.

Mathematically, R-squared is calculated as:

$R^2 = 1 – \frac{SS{res}}{SS{tot}} = \frac{SS{reg}}{SS{tot}}$

where $SS{res}$ is the residual sum of squares (the variance unexplained by the model), $SS{tot}$ is the total sum of squares (the total variance in the data), and $SS_{reg}$ is the regression sum of squares (the variance explained by the model).

R-squared ranges from 0 to 1, with higher values indicating a larger amount of variability is accounted for by the regression model. For instance, an R-squared of 0.64 means that 64% of the variance in the dependent variable can be explained by changes in the independent variables.

The Appeal of a High R-Squared

Given this definition, it seems natural to pursue a high R-squared when building regression models. After all, don‘t we want our models to explain as much of the variance in the data as possible? A model with an R-squared close to 1 appears to be nearly perfect.

This line of thinking leads many analysts down the path of what I call "R-squared hacking" – trying to add more and more independent variables to the model in an attempt to push the R-squared value as high as possible. If some variables are good, more must be better, right?

The Problem of Overfitting

Wrong. Chasing an ever-higher R-squared without regard to other considerations is a recipe for overfitting. An overfit model may have high R-squared on its training data, but its predictive performance on new, unseen data is likely to be much worse.

Overfitting happens when a model is too complex for the amount of training data available. It‘s like trying to fit a high-degree polynomial to a small number of data points – you can get a curve that passes through every point, but it will have wild oscillations in between that don‘t represent the true underlying pattern.

An example of overfitting (black curve) compared to an appropriately fit model (green curve). Source: Wikipedia

In the context of linear regression, each independent variable you add to the model increases its complexity. Even if a variable has little or no actual relationship with the outcome, including it will cause R-squared to increase slightly due to random chance alone.

For a simple demonstration, consider a simulated data set with 100 observations and 10 independent variables, where only the first variable has a real correlation with the target. Using Python with the statsmodels library, we can fit a series of models with increasing numbers of variables and track the R-squared:

import pandas as pd
import numpy as np
import statsmodels.api as sm

# Generate random data
n = 100
x1 = np.random.normal(size=n)
x2 = np.random.normal(size=n)  
x3 = np.random.normal(size=n)
x4 = np.random.normal(size=n)
x5 = np.random.normal(size=n)
x6 = np.random.normal(size=n)
x7 = np.random.normal(size=n) 
x8 = np.random.normal(size=n)
x9 = np.random.normal(size=n)
x10 = np.random.normal(size=n)

y = 2*x1 + np.random.normal(size=n)

# Fit models and store R-squared
results = []

for i in range(1, 11):
    X = np.column_stack([eval(f‘x{j}‘) for j in range(1, i+1)])
    X = sm.add_constant(X)
    model = sm.OLS(y, X).fit()
    results.append([i, model.rsquared, model.rsquared_adj])

results = pd.DataFrame(results, columns=[‘Variables‘, ‘R-squared‘, ‘Adjusted R-squared‘])
print(results)

The results show the clear trend of increasing R-squared versus a peak and decline in adjusted R-squared:

   Variables  R-squared  Adjusted R-squared
0          1   0.303194            0.296183
1          2   0.338201            0.324661
2          3   0.359582            0.339741
3          4   0.361645            0.335419
4          5   0.373846            0.341464
5          6   0.400967            0.363063
6          7   0.405946            0.362305
7          8   0.415554            0.366497  
8          9   0.415564            0.360427
9         10   0.416444            0.355275

Despite only the first variable having any real signal, R-squared increases all the way from 0.30 with one variable to 0.42 with all ten. Past about six variables though, adjusted R-squared begins to decline, correctly identifying that additional variables are not truly informative.

Adjusting for Model Complexity

As this example illustrates, the fundamental issue with using regular R-squared for model selection is that it does not take model complexity into account. This is where adjusted R-squared comes in.

Adjusted R-squared penalizes R-squared based on the number of independent variables relative to the number of data points. The formula is:

$R^2{adj} = 1 – \frac{(1-R^2)(n-1)}{n-p-1} = 1 – \frac{SS{res}/(n-p-1)}{SS_{tot}/(n-1)}$

where $n$ is the number of observations and $p$ is the number of independent variables.

Increasing the number of variables causes the second term to grow, which decreases adjusted R-squared unless the additional variables improve the fit enough to compensate. You can think of it like a cost-benefit calculation – adding variables comes at a cost in degrees of freedom, so they need to pull their weight in terms of explanatory power.

Choosing Between Metrics

This is not to say that R-squared is a useless metric. For a fixed set of independent variables, it provides a clear and intuitive measure of model fit. When comparing models with different numbers of variables though, adjusted R-squared is generally preferable as it captures the tradeoff between model complexity and additional explained variance.

Knowing when to use each metric comes down to your specific goals. Are you trying to:

1. Evaluate the predictive power of a single model? Use R-squared.
2. Compare goodness of fit between models with different numbers of variables? Use adjusted R-squared.
3. Select variables for a regression model? Stepwise selection methods using adjusted R-squared or other criteria like AIC or BIC are a good starting point.
4. Assess a model‘s generalization performance? Validate on held-out test data or use cross-validation.

No one metric can give the full picture. R-squared, adjusted R-squared, p-values, and others each provide complementary information. Using them together with appropriate visualization and validation techniques is the key to robust model development.

Beyond R-Squared

It‘s also worth noting some other model evaluation metrics beyond the R-squared family. The Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are two alternatives that balance model fit with complexity using slightly different cost functions.

Another class of metrics focuses on out-of-sample predictive accuracy directly, rather than in-sample goodness of fit. Cross-validated R-squared and mean squared error are common examples. These approaches split the data into training and validation sets multiple times and average the model‘s performance across the splits.

Advances in computing power have also enabled techniques like bootstrap resampling. By repeatedly sampling with replacement from the original data and refitting the model, we can estimate its stability and variability in a nonparametric way. Confidence intervals derived from bootstrapping can provide a valuable complement to classical hypothesis testing.

Minimum Sample Size

An important consideration for any regression model is the ratio of data points to independent variables. Models with too little data relative to their complexity are especially prone to overfitting.

A common rule of thumb is to have at least 10-15 observations per independent variable. So if you have a data set with 100 points, you should limit yourself to 7-10 variables to avoid overfitting.

More advanced techniques like regularization can help prevent overfitting with high-dimensional data by constraining the magnitude of coefficients. Ridge regression and lasso are two popular regularization methods.

Bias-Variance Tradeoff

The concepts of overfitting vs underfitting are closely tied to the bias-variance tradeoff in machine learning. Bias refers to error introduced by approximating a complex problem with a simple model. Variance refers to error introduced by a model‘s sensitivity to small fluctuations in the training data.

Underfit models have high bias but low variance – they don‘t capture all the relevant patterns in the data, but their predictions are relatively stable across different training sets. Overfit models have low bias but high variance – they match the training data very closely but don‘t generalize well to new data.

Relationship between model complexity, bias, and variance. Source: Wikipedia

The goal in model building is to find the sweet spot that balances bias and variance. Models should be complex enough to capture the true underlying patterns in the data, but not so complex that they fit noise and anomalies. Tuning model hyperparameters and using evaluation techniques like cross-validation can help navigate this tradeoff.

Case Study: Predicting House Prices

Let‘s look at an example of putting these principles into practice. The Ames Housing dataset contains 79 variables describing the characteristics of residential homes in Ames, Iowa. The goal is to predict the sale price of a house based on these features.

With so many potential variables, it would be easy to build an overfit model. Here‘s a simplified version of the process a data scientist might follow to avoid that:

1. Exploratory analysis: Look at the distributions of variables, correlations between them, and simple univariate relationships with the target. Identify the strongest predictors and any data quality issues.

2. Feature engineering: Create new variables that capture important domain knowledge, such as total square footage or interactions between location and size. Keep track of the number of new variables created.

3. Splitting: Separate the data into training, validation, and test sets. Only use the training data for model building and the validation data for model selection.

4. Model building: Start with a simple baseline model with core variables. Gradually add complexity by including more variables, interactions, or polynomial terms. Monitor changes in R-squared and adjusted R-squared.

5. Model selection: Choose the model with the highest adjusted R-squared on the validation set as the best fit. Alternatively, use other selection criteria like cross-validated mean squared error.

6. Assumption checking: Look at residual plots to verify that modeling assumptions are met. Check for influential points or outliers that could be skewing results.

7. Evaluation: Once a final model is selected, evaluate its performance on the held-out test set. Compare R-squared and other metrics to the validation results to check for overfitting.

This iterative approach helps systematically increase model complexity while guarding against fitting to noise. The use of train, validation, and test splits ensures that model selection is done on data not used for training, which is a more honest assessment of generalization performance.

Conclusions

The appeal of a high R-squared for regression models is understandable, but it must be balanced against the danger of overfitting. As we‘ve seen, chasing R-squared alone can lead to unnecessarily complex models that don‘t generalize well.

Adjusted R-squared is a simple yet effective way to account for model complexity in feature selection and model comparison. Using it in conjunction with other diagnostic plots, cross-validation, and domain knowledge helps find models that are accurate, stable, and interpretable.

Maximizing R-squared should not be the end goal in itself. At the end of the day, what matters most is how well the model solves the business problem it was designed for. Predictive power on unseen data is the real test.

By keeping these principles in mind and using a variety of tools to assess model fit, data scientists can find the right balance between explaining the data and avoiding costly overfit models. Regression modeling is both an art and a science – rigorous evaluation and thoughtful application of metrics like R-squared are essential to doing it well.