KNNImputer: A Robust Way to Impute Missing Values Using Scikit-Learn

The Problem of Missing Data

Missing values are an unavoidable reality when working with real-world datasets. Whether due to data entry errors, faulty sensors, skipped survey responses, or myriad other reasons, most datasets contain at least some missing data. This presents a major challenge for data scientists, since most machine learning algorithms cannot handle missing values natively.

There are a few different ways to address missing data:

1. Discard observations with missing values
2. Manually fill in missing values using domain knowledge
3. Automatically impute missing values using the information in the dataset

The first option of discarding incomplete observations often means throwing away a large percentage of the available data. Manually filling in missing data is time-consuming and not scalable to large datasets. Therefore, imputation is typically the best option for dealing with missing values in an efficient, data-driven way.

How the K-Nearest Neighbors Algorithm Works

The k-nearest neighbors (KNN) algorithm is a simple yet powerful approach that can be used for both classification and regression tasks. The key idea is that a data point is likely to have a similar target value as other points nearby in the feature space.

To make a prediction for a new data point, KNN finds the k closest points in the training set and returns the average of their target values. For classification, this is the most common class label among the neighbors. For regression, it‘s the mean target value of the neighbors.

The two main things to consider with KNN are:

1. How to measure distance between data points
2. How to choose the number of neighbors k

By default, KNN uses straight-line Euclidean distance to measure the distance between points. However, other distance metrics like Manhattan or Minkowski distance can be used instead. Choosing the optimal k is typically done through cross-validation, as detailed later.

Imputing Missing Values with KNNImputer

We can leverage the core concepts behind KNN to perform missing value imputation. The idea is to estimate each missing value using the values of neighboring points that do have data for that feature.

Scikit-learn provides a convenient KNNImputer class that does exactly this. To use it, you simply:

1. Instantiate a KNNImputer, specifying the number of neighbors k to use
2. Call the fit() method on the imputer object, passing in your input data
3. Call transform() or fit_transform() to fill in the missing values

Here‘s a basic code example:

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
X_imputed = imputer.fit_transform(X_missing)

Let‘s unpack what‘s happening step-by-step. First we import the KNNImputer class and create an instance, specifying to use the 5 nearest neighbors to fill in each missing value. Calling fit() makes the imputer learn the structure of the data. Finally, transform() returns a copy of the input data with all missing values replaced by the imputed estimates.

The fit_transform() method conveniently combines the fit and transform steps into one, which is useful when you don‘t need to reuse the imputer object.

Key Parameters of KNNImputer

While KNNImputer works quite well out-of-the-box, there are a few key parameters you should know about:

• n_neighbors: The number of neighbors to use for imputation. Default is 5, but this can be tuned.
• weights: How to weight the neighbors when computing imputed values. Can be ‘uniform‘ (default) or ‘distance‘. With ‘distance‘, closer neighbors have more influence.
• metric: What distance metric to use. Default is ‘nan_euclidean‘, a variant of regular Euclidean distance that handles missing values.

Choosing good values for these parameters is important for getting quality imputations. We‘ll discuss hyperparameter tuning more later on.

Handling Different Data Types

One limitation of KNNImputer is that it assumes all features are continuous numeric variables. It cannot directly handle categorical string data.

However, we can get around this by encoding categorical variables as numbers first. A simple approach is to use OrdinalEncoder to convert each unique category to an integer:

from sklearn.preprocessing import OrdinalEncoder

enc = OrdinalEncoder()
X_encoded = enc.fit_transform(X_original) 

The encoded X matrix can then be safely passed to KNNImputer. After imputation, the inverse transformation can restore the original categories:

X_imputed_decoded = enc.inverse_transform(X_imputed)

For high cardinality categorical variables, a one-hot encoding may work better than ordinal encoding. The important thing is to use a sensible numeric representation before imputing.

Evaluating Imputation Quality

How do we know if KNNImputer (or any imputation method) is doing a good job of estimating missing values? Since we don‘t know the ground truth values, it‘s impossible to directly measure imputation accuracy.

However, there are a few best practices for evaluating imputers:

1. Visual inspection of distributions
2. Train/test split with artificially removed values
3. Comparing model performance downstream

Plotting histograms or density plots of the imputed and original feature distributions can reveal if the imputer is preserving the overall data distribution. Major discrepancies are a red flag.

We can also create our own synthetic missing values by taking a complete subset of the data and randomly removing some values. Imputing on this dataset allows us to compare the estimated values to the actual ones we removed.

Lastly, imputation is usually a preprocessing step before applying supervised learning. So another metric is to compare the final trained model accuracy with different imputation methods. Better imputation should lead to better downstream results.

Scaling and Categorical Encoding with ColumnTransformer

We saw earlier how ordinal encoding enables KNNImputer to work with categorical variables. For optimal results, it‘s also a good idea to scale continuous features to a consistent range, since KNN relies on distances between points.

We can combine scaling of numeric features and encoding of categorical features into one step using scikit-learn‘s ColumnTransformer:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

ct = ColumnTransformer([
    (‘scale‘, StandardScaler(), numeric_features),
    (‘encode‘, OneHotEncoder(), categorical_features)
])

X_transformed = ct.fit_transform(X_original)
X_imputed = KNNImputer().fit_transform(X_transformed)

This ensures KNNImputer works with an optimally preprocessed dataset. Later, when making predictions on new data, be sure to apply these same column transformations before imputing.

Tuning KNNImputer Hyperparameters

To get the most out of KNNImputer, it‘s worth tuning the core hyperparameters, especially the number of neighbors k. Too low of a k leads to noisy imputations overly influenced by individual points. Too high of a k averages over too many dissimilar points. The ideal k is a balanced medium.

We can find the optimal k through cross-validation:

from sklearn.model_selection import GridSearchCV

param_grid = {‘n_neighbors‘: range(1, 20)}
imputer = KNNImputer()
grid_search = GridSearchCV(imputer, param_grid, cv=5)
grid_search.fit(X_missing)

print(f"Best number of neighbors: {grid_search.best_params_}")

This does a grid search over k values from 1 to 20 and reports which setting had the best cross-validation performance. The scoring parameter of GridSearchCV lets you specify a custom metric to optimize for. Options include ‘neg_mean_squared_error‘ (default), ‘r2‘, and others.

Choosing a good distance metric is also important. While the default Euclidean distance works well in most cases, sometimes Manhattan (L1) or Minkowski distances can be more robust to outliers. The metric parameter of KNNImputer lets you specify which distance function to use.

When to Use KNNImputer vs Other Imputers

KNNImputer has some notable advantages compared to common univariate imputation methods like mean and median imputation:

• Captures feature correlations to inform imputations
• Adapts to local data structures
• Makes fewer distributional assumptions
• Can return a range of plausible values, not just one estimate

So in many cases, KNNImputer will outperform simpler methods. However, there are situations where KNN imputation is less suitable:

• Very large datasets, where computing pairwise distances gets expensive
• Datasets with many features, where distance metrics become less meaningful
• Variables that are known to be independent, so using feature information doesn‘t help
• Time series data, where temporal order should be incorporated into imputations

Scikit-learn offers several other imputer options to handle these scenarios, such as IterativeImputer for multivariate imputation in large datasets and SimpleImputer for basic univariate strategies.

Conclusion

KNNImputer is a powerful tool for filling in missing values in a dataset by leveraging feature information. It‘s more flexible and robust than simple mean/median/mode imputation, and generally leads to higher-quality estimates.

To achieve optimal results with KNNImputer, remember to:

1. Scale numeric features and encode categorical ones
2. Tune the number of neighbors k through cross-validation
3. Consider different distance metrics like Manhattan or Minkowski
4. Evaluate imputation quality both visually and through downstream metrics

While not suited for every situation, KNNImputer is a great default choice for missing value imputation in many real-world datasets. Its simplicity and effectiveness make it a valuable part of the data scientist‘s toolkit.