The AI/ML Expert‘s Guide to Dealing with Missing Values

Missing values are a pervasive challenge in machine learning projects. In a survey of data scientists, 81% reported that their data cleaning and preprocessing steps take up more than 20% of their time, with missing values being a top contributing factor.[^1] Failing to properly handle missing data can lead to biased models, reduced predictive power, and invalid conclusions.

In this guide, we‘ll take a deep dive into the different types of missing data and cutting-edge techniques for handling them effectively. We‘ll cover both fundamental methods like deletion and imputation, as well as advanced approaches using deep learning. By the end, you‘ll have a robust toolkit for tackling missing values in your machine learning pipelines.

Understanding the Mechanisms of Missingness

The first step in dealing with missing values is to understand the mechanism that generated them. Missing data are classified into three types:[^2]

1. Missing Completely at Random (MCAR): The probability of a value being missing is unrelated to both observed and unobserved variables. The missing data points are a random subset of the data.

   For example, if a lab sample is accidentally dropped on the floor, or a survey respondent skips a question by mistake, those missing values would be MCAR.

2. Missing at Random (MAR): The probability of a value being missing depends on observed variables but not on the unobserved ones. There is a systematic relationship between the missingness and other measured variables.

   For example, if men are less likely to fill out an income field in a survey, the missingness of income would be MAR (dependent on the observed gender variable).

3. Missing Not at Random (MNAR): The probability of a value being missing depends on unobserved variables, including potentially the missing value itself. There is a systematic relationship between the missingness and unmeasured parameters.

   For example, if people with very high incomes are less likely to report their earnings in a survey, the missingness of income would be MNAR.

The type of missingness matters because different missing data handling methods have different assumptions. If those assumptions are violated, it can lead to biased results.[^3] Most methods assume the data are MAR or MCAR. MNAR is the most challenging to handle since the missingness depends on unobserved factors.

Deletion Methods for Missing Data

Deletion methods discard entire rows (listwise deletion) or specific observations (pairwise deletion) that contain missing values. Listwise deletion is also known as complete case analysis.

Advantages of deletion:

• Easy to implement
• Produces a complete dataset
• Unbiased if data are MCAR

Disadvantages of deletion:

• Loss of information and statistical power
• Biased estimates if data are MAR or MNAR

A simulation study found that listwise deletion produced biased regression coefficients even with only 10% missing data when the missingness was MAR.[^4] Therefore, deletion methods should only be used when the percentage of missing data is very small and the data are MCAR.

Fundamental Imputation Methods

Imputation fills in missing values with estimated ones. Here are some commonly used methods:

1. Mean/Median/Mode Imputation: Missing values are replaced with the mean (for continuous variables), median (for ordinal or skewed data), or mode (for categorical variables) of the observed values.

2. Regression Imputation: Missing values are predicted using a regression model based on the other variables. Linear regression is used for continuous variables and logistic regression for categorical ones.

3. K-Nearest Neighbors (KNN) Imputation: The k most similar complete cases are used to estimate the missing value. Similarity is defined based on a distance metric like Euclidean distance.

4. Multiple Imputation: Several plausible imputed datasets are created and the results are combined to incorporate uncertainty. A popular implementation is Multiple Imputation by Chained Equations (MICE), which performs a series of regression models where each variable with missingness is modeled based on the other variables.

Here is a comparison of these fundamental methods:

Method	Pros	Cons
Mean/Median/Mode	– Easy to implement	– Ignores relationship with other variables
	– Preserves sample size	– Underestimates variance
Regression	– Captures linear relationships	– Assumes MAR mechanism
	– Uses information from other variables	– Can cause bias if model is misspecified
KNN	– Captures complex relationships	– Requires complete cases
	– Non-parametric	– Computationally intensive
Multiple Imputation	– Accounts for imputation uncertainty	– Assumes MAR mechanism
	– Incorporates information from other variables	– Computationally intensive

A study comparing mean imputation, regression imputation, and multiple imputation found that multiple imputation produced the least biased parameter estimates, especially when the amount of missing data was high (>25%) and the missing data mechanism was MAR.[^5]

Advanced Imputation Methods

In addition to the fundamental methods, there are several cutting-edge imputation techniques from the machine learning literature:

1. Matrix Factorization: The incomplete data matrix is decomposed into low-rank matrices which are then used to estimate the missing values. Singular value decomposition (SVD) and non-negative matrix factorization (NMF) are popular choices.[^6]

2. Autoencoders: Autoencoders are a type of neural network that learn a compressed representation of the input data and then reconstruct the original input. The reconstruction error can be used as an unsupervised objective to train the network to impute missing values.[^7]

3. Generative Adversarial Imputation Networks (GAIN): GAIN consists of a generator that produces imputations and a discriminator that tries to distinguish between observed and imputed values. The generator is trained to fool the discriminator, resulting in plausible imputations.[^8]

Here is a comparison table of these advanced methods:

Method	Pros	Cons
Matrix Factorization	– Captures global structure of data	– Assumes linear relationships
	– Computationally efficient for large datasets	– Difficult to incorporate side information
Autoencoders	– Captures non-linear relationships	– Requires large amount of training data
	– Can incorporate side information	– Black box model
GAIN	– Generates plausible imputations	– Difficult to train
	– Doesn‘t require complete cases	– Computationally intensive

A recent benchmark study compared matrix factorization, autoencoders, GAIN, and MICE on several real-world datasets.[^9] They found that GAIN and autoencoders outperformed MICE and matrix factorization when the missing rate was high (>50%) and the data contained non-linear relationships. However, training GAINs was significantly more computationally expensive.

Assessing Imputation Performance

To evaluate the performance of an imputation method, we need to assess the similarity between the imputed values and the ground truth. Common evaluation metrics include:

• Normalized root mean squared error (NRMSE): Measures the average difference between the imputed and actual values, normalized by the range of the data.

• Proportion of falsely classified entries (PFC): For categorical variables, PFC is the proportion of cases where the imputed class doesn‘t match the true class.

However, we usually don‘t know the ground truth for the missing values. One solution is to simulate missingness on a complete dataset by randomly removing values and then comparing the imputations to the actual removed values.[^10] This can help assess the performance of different imputation methods under controlled settings.

It‘s also important to evaluate the impact of imputation on downstream modeling tasks. The best imputation method is one that results in the best performance on the final predictive model. Therefore, it‘s recommended to compare several imputation methods and choose the one that gives the highest cross-validated accuracy or lowest error on a holdout test set.

Best Practices for Missing Data Handling

Here are some general tips for dealing with missing values in machine learning projects:

1. Examine the patterns of missingness: Visualize the distribution of missing values across features and samples. Look for any systematic patterns that could indicate the missing data mechanism.

2. Investigate potential causes: Use domain expertise to hypothesize why certain values could be missing. Is it due to faulty sensors, data entry errors, or study dropouts? Understanding the cause can help inform the imputation approach.

3. Consider the analysis goal: The best imputation method depends on the downstream task. For inference, you may want to use multiple imputation to propagate uncertainty. For prediction, you may prioritize computational efficiency.

4. Compare multiple methods: There‘s no one-size-fits-all imputation method. Try several approaches and compare their performance on relevant metrics. Use cross-validation to avoid overfitting.

5. Document your process: Keep detailed records of how you handled missing data, including the percentage of missing values, assumed missing data mechanism, imputation methods used, and evaluation results. This is crucial for reproducibility and understanding limitations of the analysis.

6. Perform sensitivity analysis: Investigate how sensitive your results are to different imputation approaches. Do the conclusions change significantly if you use mean imputation vs multiple imputation? If so, that indicates the need for caution in interpreting the results.

Industry-Specific Imputation Techniques

The best imputation approach also depends on the specific domain and data type. Here are some examples of industry-specific techniques:

• Healthcare: In electronic health records, lab tests and vital signs are often missing not at random (e.g. sicker patients may get more tests). Imputation methods that rely on the MAR assumption can lead to biased results. A promising approach is to use a mixture model that simultaneously models the missing data mechanism and performs imputation.[^11]

• Finance: Financial time series data often contain missing values due to non-trading days or data recording gaps. Interpolation methods that capture trends, seasonality, and regime changes are well-suited for this type of data. Examples include state space models, Kalman smoothing, and regime-switching models.[^12]

• Natural Language Processing: When dealing with missing words or phrases in text data, imputation techniques need to capture the semantic context. Neural language models like BERT can be fine-tuned for this task by masking random words and training the model to predict the masked tokens.[^13]

• Computer Vision: In image datasets, missing values can occur due to occlusion, corruption, or data augmentation. Inpainting techniques based on Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) can synthesize plausible imputations that blend seamlessly with the surrounding pixels.[^14]

The key is to leverage domain knowledge to inform the imputation process. Collaborate with subject matter experts to understand potential causes of missingness, validate imputed values, and interpret results.

Conclusion

Dealing with missing values is a critical skill for any data scientist or machine learning practitioner. By understanding the different types of missing data mechanisms and appropriate handling techniques, you can mitigate bias, improve model performance, and extract valuable insights from incomplete data.

To summarize the key takeaways:

• Determine the missing data mechanism (MCAR, MAR, or MNAR) to guide imputation method selection
• Use deletion methods only if missing rate is very low and data are MCAR
• For MAR data, consider regression imputation, KNN imputation, or multiple imputation
• Advanced methods like matrix factorization, autoencoders, and GAIN can capture complex patterns but are computationally intensive
• Assess imputation performance using both statistical metrics and impact on downstream modeling tasks
• Document missing data handling process for reproducibility
• Leverage domain expertise to inform imputation and validate results

As a final thought, while imputation can help salvage incomplete datasets, the best solution is to prevent missing data in the first place. Work with data collection teams to improve processes, use validated measurement tools, and follow up on missing values. High-quality, complete data is the foundation of successful machine learning projects.

[^1]: Anaconda. (2020). State of Data Science 2020. https://www.anaconda.com/state-of-data-science-2020

[^2]: Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592.

[^3]: Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the art. Psychological Methods, 7(2), 147-177.

[^4]: Pepinsky, T. B. (2018). A note on listwise deletion versus multiple imputation. Political Analysis, 26(4), 480-488.

[^5]: Horton, N. J., & Lipsitz, S. R. (2001). Multiple imputation in practice: comparison of software packages for regression models with missing variables. The American Statistician, 55(3), 244-254.

[^6]: Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30-37.

[^7]: Gondara, L., & Wang, K. (2018). Mida: Multiple imputation using denoising autoencoders. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 260-272).

[^8]: Yoon, J., Jordon, J., & Van Der Schaar, M. (2018). GAIN: Missing data imputation using generative adversarial nets. arXiv preprint arXiv:1806.02920.

[^9]: Camino, R. D., Hammerschmidt, C. A., & State, R. (2019). Improving missing data imputation with deep generative models. arXiv preprint arXiv:1902.10666.

[^10]: Twala, B., Jones, M., & Hand, D. J. (2008). Good methods for coping with missing data in decision trees. Pattern Recognition Letters, 29(7), 950-956.

[^11]: Ibrahim, J. G., Chu, H., & Chen, M. H. (2012). Missing data in clinical studies: issues and methods. Journal of Clinical Oncology, 30(26), 3297.

[^12]: Borman, P. (2009). Interpolating gaps in time series using state space models. Econometric Reviews, 28(1-3), 291-309.

[^13]: Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

[^14]: Li, H., Li, J., Wang, S., Wu, X., & Xin, S. (2020). Image inpainting based on generative adversarial networks. IEEE Access, 8, 96661-96668.