Handling Missing Values with Random Forest Imputation

Missing data is a common challenge faced in almost every real-world data analysis project. Data can be missing for a variety of reasons – survey questions left blank, faulty sensors, data entry errors, and so on. Regardless of the cause, missing values can be problematic for many machine learning algorithms that expect complete data as input. Improper handling of missing data can lead to biased parameter estimates and skewed results.

While it may be tempting to simply remove any observations with missing values, this is often not advisable as it reduces sample size and statistical power. A better approach is to impute, or fill in, the missing values using information from the non-missing data. There are many different imputation techniques that can be employed, ranging from simple methods like mean/median imputation to more sophisticated machine learning based approaches.

In this post, we‘ll take a deep dive into using random forests for missing value imputation. Random forest imputation has emerged as a powerful technique that offers several advantages over other common imputation methods. We‘ll explain how it works, walk through some code examples in Python, and discuss best practices for using random forests to handle missing data in your own projects.

A Brief Primer on Missing Data

Before we jump into the details of random forest imputation, let‘s briefly review what missing data is and why it poses challenges for data analysis. Missing data refers to values that are not present in a dataset for one or more variables. There are three main mechanisms that give rise to missing data:

1. Missing completely at random (MCAR): The probability of a value being missing is unrelated to both the observed and unobserved data. In other words, there is no systematic reason for the missingness. An example would be a survey respondent accidentally skipping a question.

2. Missing at random (MAR): The probability of a value being missing depends only on the observed data, not the missing values themselves. For instance, men may be less likely than women to answer a question about income, but within each gender, the missingness is random.

3. Missing not at random (MNAR): The probability of a value being missing depends on the unobserved value itself, even after accounting for observed data. For example, individuals with very high incomes may be less inclined to report their earnings on a survey.

Most imputation methods, including random forest imputation, assume the data are either MCAR or MAR. If the data are MNAR, imputation can be much more challenging and may require specialized techniques beyond the scope of this article.

The primary issue with missing data is that many machine learning algorithms cannot handle observations with missing values. Rows containing missing data may be excluded entirely from model training and evaluation, leading to loss of information. Improperly handling missing values can also introduce bias into model estimates and predictions.

Overview of Imputation Methods

Imputation is the process of replacing missing values with substituted values. The goal is to produce a complete dataset that can then be used for downstream analysis and modeling. There are a variety of imputation methods that have been proposed, each with its own strengths and weaknesses. Some common techniques include:

• Mean/median imputation: Missing values are replaced with the mean (for continuous variables) or mode (for categorical variables) of the non-missing values. While simple to implement, this method can distort the distribution of the data, especially if there are a large number of missing values.

• Last observation carried forward (LOCF): Missing values are replaced with the last observed value for that variable. This is often used in longitudinal studies with repeated measurements over time, but can introduce bias if the missingness is related to the outcome of interest.

• Regression imputation: Missing values are predicted using a regression model fit on the non-missing data. The model can include other variables in the dataset as predictors. However, this can underestimate the variance in the data.

• Stochastic regression imputation: Similar to regression imputation, but random noise is added to the predicted values to better preserve the variability in the data.

• Hot deck imputation: Missing values are replaced with observed values from "similar" observations in the dataset, where similarity is defined based on other variables.

• K-nearest neighbors (KNN) imputation: Missing values are replaced with the average of the K closest observations, where distance is calculated in the space of the other variables. This allows for imputation based on localized information, but can be sensitive to the choice of K.

• Multiple imputation by chained equations (MICE): A series of regression models are fit for each variable with missing data, using the other variables as predictors. This process is repeated iteratively to refine the imputations. The final imputed values are drawn from the regression predictions, allowing for uncertainty in the imputations.

While these methods can work well in some cases, they also have limitations. Many rely on strong parametric assumptions or are ill-equipped to handle high-dimensional, mixed-type data. This is where random forest imputation comes in.

Random Forest Imputation

Random forest imputation is a nonparametric, machine learning based approach for handling missing data. It has gained popularity in recent years due to its flexibility, scalability, and ability to capture complex relationships between variables.

At a high level, the idea behind random forest imputation is to treat the variable with missing values as the outcome to be predicted, and use the other variables in the dataset as predictors. A random forest model is fit on the observations with complete data, and then used to predict the missing values. This process is repeated iteratively, with the imputed values updated at each iteration based on the current model predictions.

There are a few key advantages to using random forests for imputation:

1. Nonparametric: Random forests do not make any assumptions about the functional form of the relationships between variables. This allows them to capture complex, nonlinear associations that may be missed by other methods.

2. Mixed-type data: Random forests can easily handle a mix of continuous and categorical variables without the need for explicit data transformation. The splitting criteria used to grow the trees adapt to the type of data being modeled.

3. High-dimensional data: Random forests are well-suited for datasets with a large number of variables relative to the number of observations. The random subsampling of variables at each split helps prevent overfitting and allows for efficient scaling to high dimensions.

4. Automatic variable selection: The recursive partitioning process used to grow the trees naturally incorporates a form of variable selection. Variables that are most informative for predicting the outcome will be selected more frequently for splitting, while less relevant variables will be largely ignored. This helps the imputation process focus on the most important information.

5. Robustness to outliers: Random forests are generally robust to the presence of outliers or extreme values in the data. The final predictions are made by averaging across a large number of trees, which helps smooth out the influence of any individual anomalous observations.

Miss Forest Algorithm

One of the most popular random forest imputation algorithms is Miss Forest, proposed by Stekhoven and Bühlmann in 2012. The Miss Forest algorithm follows these main steps:

1. Initial imputation: Missing values are initially imputed using simple techniques like mean or median imputation. This provides a starting point for the iterative imputation process.

2. Iterative imputation: For each variable with missing values, a random forest model is fit using the other variables as predictors. The model is then used to predict the missing values for that variable. This process is repeated for each variable, cycling through the variables multiple times until convergence.

3. Prediction averaging: The final imputed values are calculated by averaging the predictions across all the trees in the random forest.

One key hyperparameter in the Miss Forest algorithm is the number of iterations to perform. Generally, a small number of iterations (e.g. 5-10) is sufficient for convergence. Increasing the number of iterations can improve imputation accuracy, but at the cost of increased computation time.

Mice Forest Algorithm

Another popular variant of random forest imputation is the MICE Forest algorithm, proposed by Liu et al. in 2020. MICE Forest combines the ideas of random forest imputation with the multiple imputation by chained equations (MICE) framework.

The key steps in the MICE Forest algorithm are:

1. Initial imputation: As with Miss Forest, missing values are initially imputed using simple techniques to provide a starting point.

2. Multiple imputation: Instead of a single imputation, MICE Forest generates multiple imputed datasets. This allows for the incorporation of imputation uncertainty into downstream analyses.

3. Chained equations: For each variable with missing values, a random forest model is fit using the other variables as predictors. However, instead of directly predicting the missing values, the model is used to generate predictions for all observations (both missing and non-missing). These predictions serve as the updated imputations for that variable.

4. Iteration: The chained equations process is repeated for a fixed number of iterations, with the imputations updated at each iteration based on the current predictions.

5. Pooling: The final imputed values are calculated by pooling the predictions across all the imputed datasets, typically using simple averaging.

The multiple imputation aspect of MICE Forest allows for the assessment of imputation uncertainty and can lead to more robust inferences in downstream analyses. However, it also requires more computation time and storage compared to Miss Forest, as multiple imputed datasets need to be generated and managed.

Python Examples

To illustrate random forest imputation in action, let‘s walk through some examples using Python. We‘ll use the popular scikit-learn library for implementing the random forest models, and the missingpy library for the Miss Forest algorithm.

First, let‘s generate a simple dataset with missing values:

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing

data = fetch_california_housing(as_frame=True)[‘frame‘]
data = data.drop(columns=[‘MedInc‘]) # Remove target variable

# Introduce missing values (MCAR)
mask = np.random.rand(*data.shape) < 0.1
data_missing = data.mask(mask)

Here we‘ve used the California Housing dataset and randomly masked 10% of the values to simulate MCAR missingness.

Next, let‘s impute the missing values using the Miss Forest algorithm:

from missingpy import MissForest

imputer = MissForest(max_iter=10, n_estimators=100, verbose=1)
data_imputed = imputer.fit_transform(data_missing)

We‘ve specified a maximum of 10 iterations and 100 trees per forest. The verbose parameter provides progress updates during the imputation process.

For comparison, let‘s also impute the missing values using KNN imputation:

from sklearn.impute import KNNImputer

imputer_knn = KNNImputer(n_neighbors=5)
data_imputed_knn = imputer_knn.fit_transform(data_missing)

We can evaluate the imputation performance by calculating the mean squared error (MSE) between the imputed values and the true values (which we know since we artificially introduced the missingness):

from sklearn.metrics import mean_squared_error

mse_mf = mean_squared_error(data[mask], data_imputed[mask])
mse_knn = mean_squared_error(data[mask], data_imputed_knn[mask])

print(f"Miss Forest MSE: {mse_mf:.3f}")
print(f"KNN MSE: {mse_knn:.3f}")

In this example, the Miss Forest algorithm achieves a lower MSE than KNN imputation, indicating better imputation accuracy.

We can also visualize the imputed values compared to the true values using scatter plots:

import matplotlib.pyplot as plt

fig, axs = plt.subplots(1, 2, figsize=(12, 6))
axs[0].scatter(data[mask], data_imputed[mask], alpha=0.5)
axs[0].set_title("Miss Forest")
axs[0].set_xlabel("True Value")
axs[0].set_ylabel("Imputed Value")
axs[1].scatter(data[mask], data_imputed_knn[mask], alpha=0.5)
axs[1].set_title("KNN")
axs[1].set_xlabel("True Value")
plt.tight_layout()
plt.show()

The scatter plots show that the Miss Forest imputations more closely match the true values compared to KNN imputation, which tends to underestimate the larger values.

Limitations and Considerations

While random forest imputation is a powerful and flexible technique, it‘s important to keep in mind a few limitations and considerations:

1. Computation time: Random forest imputation can be computationally intensive, especially for large datasets or a high proportion of missing values. The iterative nature of the algorithm means that multiple random forest models need to be fit, which can be time-consuming.

2. Imputation uncertainty: Like any imputation method, random forest imputation introduces additional uncertainty into the data. It‘s important to account for this uncertainty when drawing conclusions from downstream analyses, such as by using multiple imputation techniques like MICE Forest.

3. Missingness mechanism: Random forest imputation assumes the data are missing at random (MAR). If the missingness is actually due to a systematic, nonrandom cause (i.e. MNAR), the imputations may be biased. It‘s important to carefully consider the potential missingness mechanisms in your data before applying imputation.

4. Overfitting: While random forests are generally robust to overfitting, it‘s still possible for the imputation model to overfit to the observed data, especially if there are a large number of variables relative to the number of observations. It‘s a good idea to tune the hyperparameters of the random forest (e.g. number of trees, maximum depth) to find a good balance between imputation accuracy and generalizability.

5. Multicollinearity: Random forest imputation can handle moderate multicollinearity between variables, but may struggle with extreme multicollinearity (i.e. perfect linear relationships). In these cases, the imputation model may have difficulty distinguishing the individual effects of the correlated variables, leading to instability in the imputations.

Conclusion

Missing data is a common challenge in real-world data analysis, and improper handling of missing values can lead to biased and unreliable results. Random forest imputation has emerged as a powerful technique for imputing missing values, offering several advantages over traditional methods. By leveraging the flexibility and robustness of random forests, missing values can be effectively imputed even in complex, high-dimensional datasets.

In this post, we‘ve covered the basics of missing data and imputation, and taken a deep dive into random forest imputation. We‘ve explored two popular random forest imputation algorithms – Miss Forest and MICE Forest – and walked through examples of how to implement them in Python. Finally, we‘ve discussed some limitations and considerations to keep in mind when applying random forest imputation in practice.

While random forest imputation is not a silver bullet for all missing data problems, it is a valuable tool to have in your data science toolkit. When used appropriately and with careful consideration of the assumptions and limitations, random forest imputation can help you extract valuable insights from incomplete data and improve the reliability of your analyses.