Mastering Missing Value Imputation: Top 5 Interview Questions for Machine Learning Practitioners

Missing data is ubiquitous in real-world datasets, presenting a major challenge for machine learning models that typically require complete input data. Thoughtful handling of missing values through imputation techniques is therefore a critical skill for any data scientist or ML engineer.

In this post, we‘ll dive deep into the top 5 interview questions on missing value imputation, exploring the intuition, assumptions, and practical considerations behind different imputation strategies. Whether you‘re preparing for ML interviews or seeking to level up your data preprocessing skills, this guide will equip you with an expert-level understanding of this important topic.

1. What is complete case analysis (CCA) and when is it appropriate to use?

Complete case analysis, also known as listwise deletion, is perhaps the simplest approach to handling missing data. With CCA, we simply remove any samples (rows) that contain one or more missing values, keeping only the "complete cases."

CCA has a few notable advantages:

• Straightforward to implement (a one-liner in pandas with df.dropna())
• Preserves the distribution of the data (no imputation to introduce bias)
• Useful when missingness is rare and sample size is large

However, CCA also has significant drawbacks:

• Deletes potentially useful information along with the missing data
• Reduces effective sample size, limiting the model‘s ability to learn
• Produces biased estimates if missingness is related to the outcome or features

Crucially, CCA is only statistically valid under the strong assumption that the data is missing completely at random (MCAR). MCAR means the probability of a value being missing is independent of both the observed and unobserved data. In practice, this is a high bar – missingness is often related to other features or the target variable.

A classic example of MCAR is a weighing scale that runs out of batteries, leading to randomly missing weight measurements. But if heavier individuals are more likely to decline to be weighed, the data would be missing at random (MAR) or missing not at random (MNAR), and CCA would produce biased results.

According to seminal work by Little and Rubin[^1], CCA is not recommended unless the percentage of cases with missing data is below 5% and there is strong evidence that the missingness mechanism is MCAR. Otherwise, imputation methods that are valid under MAR should be preferred.

[^1]: Little, R.J. and Rubin, D.B., 2019. Statistical analysis with missing data (Vol. 793). John Wiley & Sons.

2. For numerical data with outliers, is mean or median imputation more appropriate?

Mean and median imputation are classic univariate techniques for handling missing values in numerical data. Mean imputation fills in missing values with the average of the observed values for that feature, while median imputation uses the middle value.

In the presence of outliers – extreme values far from the center of the distribution – median imputation is generally more robust. To see why, consider this small dataset:

values = [1, 2, 2, 3, 4, 100]  # 100 is an outlier

The mean of these values is 18.7, much higher than the "typical" value due to the influence of the outlier. The median, however, is 2.5 – much closer to the central tendency of the data.

Imputing with the mean would fill in missing values with an unusually high number, while imputing with the median would produce more reasonable values. This is backed up by statistical theory: the median has a breakdown point of 50%, meaning it can tolerate up to 50% of the data being outliers before giving an arbitrarily bad result[^2].

[^2]: Huber, P.J. and Ronchetti, E.M., 2009. Robust statistics (Vol. 523). John Wiley & Sons.

However, neither mean nor median imputation is a panacea for outliers. They are still univariate methods that don‘t account for relationships between features. If the percentage of outliers is high, it‘s worth investigating why those extreme values exist and considering more advanced imputation techniques that can handle them, like winsorization or multiple imputation.

3. Contrast univariate and multivariate imputation, providing examples of each.

Imputation techniques broadly fall into two camps: univariate methods, which only consider a single feature when filling in its missing values, and multivariate methods, which use information from multiple features.

Common univariate imputation techniques include:

• Mean/median imputation
• Most frequent category imputation (mode)
• Constant value imputation (e.g. "Missing" for categorical data, 0 for numerical)
• Stochastic regression imputation

For example, we can easily perform median imputation in scikit-learn:

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy=‘median‘)
X_imputed = imputer.fit_transform(X_missing) 

Univariate techniques have the advantage of simplicity, but they fail to capture relationships between features and can bias downstream analyses by reducing variance or generating "impossible" combinations of values.

Multivariate imputation, on the other hand, leverages patterns across features to fill in missing data. Leading multivariate techniques include:

• K-Nearest Neighbors (KNN) imputation
• Iterative imputation (e.g. MICE, missForest)
• Matrix factorization (e.g. SoftImpute, LRMC)
• Deep learning imputation (e.g. MIDA, GAIN)

For instance, scikit-learn offers multivariate KNN imputation:

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)
X_imputed = imputer.fit_transform(X_missing)

Multivariate methods can often achieve lower imputation errors by learning from the observed data, but they tend to be more computationally intensive and make stronger assumptions about the missingness mechanism and the relationships between features.

A recent benchmark study[^3] on 15 real-world datasets found that multivariate methods (especially MICE and missForest) consistently outperformed univariate techniques in terms of imputation quality and downstream predictive accuracy, with the gap widening as the percentage of missing data increased.

[^3]: Jadhav, A., Pramod, D. and Ramanathan, K., 2019. Comparison of performance of data imputation methods for numeric dataset. Applied Artificial Intelligence, 33(10), pp.913-933.

4. How do KNN imputation and iterative imputation work, and what distinguishes them?

KNN and iterative imputation are two popular multivariate techniques that take quite different approaches to filling in missing values.

KNN Imputation:
The intuition behind KNN imputation is straightforward: samples that are "close" in the feature space should have similar values, even for features with missingness. For each sample with a missing value, KNN finds its K nearest neighbors using a distance metric (often Euclidean), then fills in the missing value with the mean or mode of the corresponding feature values from those neighbors.

Some pros and cons of KNN imputation:

• Makes few assumptions about the distribution of the data
• Effective for datasets with non-linear relationships between features
• Sensitive to the choice of K and the distance metric
• Struggles with high-dimensional data and nominal categorical features

Iterative Imputation:
Iterative imputation, typified by techniques like MICE (Multivariate Imputation by Chained Equations), takes a different approach – it treats missing data points as parameters to be estimated in a joint model.

The procedure works as follows:

1. Initialize the missing values (e.g. with simple mean imputation)
2. For each feature with missingness:
   • Set the observed values as the target
   • Fit a regression model on the other features
   • Impute the missing values using the model‘s predictions
3. Repeat step 2 for several iterations or until convergence

Thus, iterative imputation sequentially models each feature as a function of the others, leveraging the interrelationships to refine its estimates at each step. Some key features of this approach:

• Preserves multivariate relationships and handles mixed data types
• Allows for customization of the regression models for each feature
• Can be computationally intensive, especially with many features
• Requires careful specification of the models to avoid overfitting

A variant of iterative imputation called missForest[^4] uses random forests as the base estimator, which can automatically capture complex non-linear relationships and interactions.

In extensive benchmarks, iterative imputation methods like MICE and missForest have consistently achieved state-of-the-art imputation quality and downstream task performance[^3][^5]. They are particularly effective under MAR and MNAR mechanisms.

[^4]: Stekhoven, D.J. and Bühlmann, P., 2012. MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1), pp.112-118.
[^5]: Luo, Y., Szolovits, P., Dighe, A.S. and Baron, J.M., 2018. 3D-MICE: integration of cross-sectional and longitudinal imputation for multi-analyte longitudinal clinical data. Journal of the American Medical Informatics Association, 25(6), pp.645-653.

5. What assumptions underlie KNN and iterative imputation, and when are they most suitable?

Both KNN and iterative imputation rely on several key assumptions to produce statistically valid imputations:

KNN Assumptions:

1. The missingness mechanism is ignorable (MCAR or MAR). If missingness depends on unobserved values (MNAR), KNN can yield biased results.

2. The feature space is appropriate for calculating distances. This requires thoughtful feature scaling and encoding of categorical variables.

3. The data has sufficient local density (i.e. enough complete cases among the nearest neighbors). If the data is too sparse, the imputed values will be heavily influenced by a few points.

Thus, KNN tends to work best when:

• The MAR assumption is reasonable
• Distances in feature space correlate with similarity of the target values
• The data is not extremely high-dimensional (KNN‘s performance degrades in high dimensions)
• There is sufficient local density to support the imputation

Iterative Imputation Assumptions:

1. Again, MCAR or MAR is required for unbiased estimates. Most implementations of MICE and missForest assume MAR.

2. The specified conditional models (e.g. linear regression for continuous features, logistic regression for binary) are appropriate for the data and can capture the relevant relationships between features.

3. The cycling of imputations will converge to the joint distribution of the data. This requires careful selection of the number of iterations and the order of imputation across features.

Iterative methods like MICE are a good choice when:

• The MAR assumption is plausible, and MCAR is doubtful
• There are informative relationships between features that can aid in imputation
• Computational efficiency is not a major concern
• There is enough domain knowledge to specify appropriate conditional models (or a method like missForest is used to automate this)

Notably, both KNN and iterative methods have several tuning parameters (e.g. the number of neighbors K, the number of iterations, the regression models) that can significantly impact performance. Techniques like cross-validation should be used to select these hyperparameters and evaluate imputation quality before downstream modeling.

An emerging line of research applies deep learning to imputation, using architectures like denoising autoencoders and generative adversarial networks to learn complex representations of the data and handle MNAR mechanisms[^6]. While promising, these techniques are still relatively new and computationally demanding.

Ultimately, the choice of imputation method should be guided by the characteristics of the data (dimensionality, sparsity, missingness percentage), the assumed missingness mechanism (MCAR, MAR, MNAR), computational constraints, and the goals of the downstream analysis. A robust pipeline will often compare multiple imputation strategies and assess their impact on the task at hand.

[^6]: Mattei, A. and Frellsen, J., 2019, April. MIWAE: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Learning (pp. 4413-4423). PMLR.

Conclusion

Missing data is a pervasive challenge in machine learning, and effectively handling it through imputation is a key skill for practitioners. This post has covered five critical questions that often arise in interviews and in practice:

1. When is complete case analysis appropriate? (Almost never!)
2. How do mean and median imputation handle outliers differently?
3. What‘s the difference between univariate and multivariate imputation?
4. How do popular multivariate methods like KNN and iterative imputation work?
5. What assumptions underlie these methods, and when are they most suitable?

We‘ve seen that while simple methods like CCA and mean/median imputation can occasionally suffice, modern multivariate techniques like KNN, MICE, and missForest are usually more statistically principled and empirically effective, especially under MAR missingness. However, each method has its own assumptions, strengths, and weaknesses that must be carefully considered in the context of the problem at hand.

Looking forward, the field of missing data imputation is rapidly evolving, with deep learning approaches showing promise for tackling complex MNAR scenarios. As a practitioner, staying up to date with these developments and honing your skills in evaluating and applying imputation techniques will be invaluable.

Remember, imputation is just one step in the end-to-end machine learning process – but it‘s a crucial one. By understanding the landscape of imputation methods and their underlying statistical principles, you‘ll be well-equipped to handle missing data in your projects and discussions. Here‘s to data completeness and successful modeling!