Data Validation vs Data Verification: A Machine Learning Perspective

Data is often called the new oil fueling the AI revolution. But just like oil, data in its raw form has limited value – it needs to be processed, refined, and quality controlled before it can power machine learning models. That‘s where data validation and data verification come in.

As an AI and machine learning expert, I‘ve seen firsthand how poor data quality can derail even the most promising ML projects. In one Gartner survey, companies reported that 85% of AI projects ultimately fail to deliver on their intended promises to business. Data issues are a common culprit. The old adage "garbage in, garbage out" is especially true for machine learning, where models will happily learn the wrong things from bad training data.

Rigorous data validation and verification throughout the ML lifecycle is critical for ensuring only high quality data is used to develop models and that models continue to perform as intended in production. In this article, we‘ll take a deep dive into data validation and verification techniques and best practices at each phase of the machine learning workflow.

Data Validation and Verification in the ML Workflow

A typical machine learning project follows this high-level workflow:

1. Problem definition and project planning
2. Data acquisition and exploration
3. Data validation and cleaning
4. Feature engineering
5. Model training and tuning
6. Model evaluation and verification
7. Model deployment to production
8. Model monitoring and maintenance
9. Data and model updating

Let‘s look at where data validation and verification fit in each stage:

Problem Definition and Project Planning

It‘s never too early to start thinking about data quality. When scoping a new machine learning project, consider:

• What data sources will be needed? How reliable and consistent are they?
• How much historical data is available for training and validation?
• What are the risks and impact of bad data or an incorrect model on the business?
• What data validation and model verification approaches will be used?

Documenting a data validation and model verification plan upfront, as part of the project proposal, helps ensure adequate time and resources are allocated to these critical tasks.

Data Acquisition and Exploration

Once the initial data is collected, preliminary data validation should be done during exploratory data analysis (EDA). Key things to check at this stage include:

• Data completeness: Check for missing data and decide how to handle it (e.g. imputation, deletion)
• Data accuracy: Spot check some samples to verify the data accuracy. But full verification often needs to wait until the data is cleaned and merged.
• Data consistency: Look for inconsistencies within and between data sources, e.g. the same field with different units of measurement.
• Data timeliness: Validate that the data covers the necessary time range and is up-to-date relative to the business use case.
• Data relevance: Verify that the available data is actually relevant to the target prediction problem. Irrelevant data is just noise.

IBM estimates that poor data quality costs the US economy $3.1 trillion per year. Finding data issues early, before too much work is invested, helps avoid wasted effort.

Data Validation and Cleaning

After EDA, the next step is in-depth data validation and cleaning to make the data ready for modeling. Depending on the data types and domain, validation at this stage may include:

• Schema validation: Programmatically checking that the data conforms to the expected schema (field names, types, value ranges, etc.). Open source libraries like Cerberus and schema allow you to define schemas in code and validate data against them.

• Data quality checks: Validating data distributions, outliers, missing values, duplicates, etc. against the expectations set during EDA. For example, verifying that a "price" field is always non-negative. Packages like Deequ and Great Expectations automate data validation and profiling.

• Cross-field validation: Checking for logical consistencies between fields. For instance, for an ecommerce dataset, verifying TotalPrice = ItemPrice * Quantity for each transaction.

• Data cleaning: Fixing or removing invalid, inaccurate, or irrelevant data identified during validation. Data cleaning must be done carefully to avoid introducing bias. All transformations should be justified and logged.

In the Kaggle 2021 State of Data Science survey, data scientists reported spending 45% of their time on data loading and cleansing tasks. Automating data validation with tools like Deequ can significantly improve productivity.

Feature Engineering

As new features are engineered, they should be individually validated using similar automated data quality checks as the raw input data. Crucially, verify that the feature values match the business understanding.

For example, when creating a "customer lifetime value" feature, validate that the calculated values are within reasonable bounds and match the team‘s domain knowledge. If the feature has nonsensical values, revisit the implementation before using it for modeling.

Model Training and Validation

With the engineered features ready, it‘s time to train and validate ML models. Careful validation of model performance on holdout test data is essential. Some key model validation approaches:

• Simple train/test split: Randomly split data into separate training and testing sets. Train model only on the training set, and validate performance on the unseen test set. This is the most basic model validation method.

• Cross-validation: Randomly split data into K "folds", train and validate model K times, each time using a different fold as the validation set. Provides more robust performance estimates, especially for smaller datasets.

• Stratified sampling: For imbalanced classification problems, it‘s important that the train/test split or cross-validation folds have equal representation of the minority class. Stratified sampling does controlled random sampling to ensure each subset has the same class balance as the full dataset.

• Time series splits: For time series problems like demand forecasting, simple random splits are not valid since they can leak future information. Instead, split data into training and validation sets by time, and validate the model on a later time period than it was trained on.

The choice of model validation approach depends on the prediction problem and data characteristics. Using an inappropriate validation method can lead to overly optimistic performance estimates and model overfitting.

Model Verification and Deployment

Before pushing an ML model to production, it‘s critical to verify it meets the original business objectives and will perform as intended in the real world. Model verification steps include:

• Holdout validation: Evaluate the final trained model on a never-before-seen test set to get an unbiased estimate of real-world performance.
• Slice-based validation: Validate model performance on important data subsets (slices). For example, check model performance across different geographic regions, customer segments, or product categories.
• Stress testing: Verify model behavior on edge cases and outliers. For instance, validate how a fraud detection model handles a sudden 10x spike in transaction volume.
• Shadow testing: Deploy the model in "shadow mode" to production, where it receives real input data but its predictions are not used. Compare the shadow model‘s predictions to the current production model to validate performance.
• A/B testing: When replacing an existing model with a new one, first route a small fraction of the live traffic to the new model and verify that it performs at least as well as the old model before rolling it out fully.

Thorough verification before deployment is essential, especially for high stakes applications like healthcare or financial models. But verification shouldn‘t stop at deployment…

Model Monitoring and Maintenance

ML models can degrade in production over time as data drifts and the real world changes. Continuous monitoring and verification is needed to track live performance and spot issues before they impact the business. Key things to monitor:

• Data drift: Detect changes in the distribution of the input data that may affect model accuracy. Tools like Evidently and WhyLogs can automatically calculate drift statistics.

• Prediction drift: Track the distribution of model predictions over time. A sudden change in predictions can indicate an issue somewhere in the data pipeline.

• Model decay: Benchmark the live model‘s predictions against known reference data (e.g. historical data) to track decay in accuracy due to data drift or model staleness.

• Data quality: Constantly validate input data against expected schema and distributions. If data validation fails, alert the data science and data engineering teams.

• Computational performance: Monitor model latency and resource utilization. A model that is too slow or resource-intensive can harm the user experience and increase cloud costs.

Monitoring enables proactive maintenance. If data or performance drifts too far, the model may need to be retrained, or the data pipelines fixed.

Data and Model Updating

Models are not static – they need to be continually updated with new training data to maintain accuracy as the world changes. When adding new data to an existing model, proper validation is essential to avoid model pollution:

• Data validation: Validate that the new data matches the schema and distributions of the original training data. If the new data is too different, it may introduce bias or reduce model performance.

• Data consistency: Ensure the new data is consistent with existing data (e.g. same units, feature definitions) before merging it in. Inconsistent data will harm the model.

• Incremental validation: When updating a model with new data, first validate it on a copy of the original model. If performance degrades, investigate before retraining.

• Continuous delivery: Adopt DevOps practices like continuous integration and delivery (CI/CD) to automatically validate data and models as part of the updating workflow. CI/CD helps catch issues early.

Challenges in Data Validation and ML Verification

While the techniques we‘ve discussed can catch many issues, data validation and ML verification is still challenging for several reasons:

• Data volume and variety: ML models often rely on huge volumes of diverse, unstructured data. Exhaustively validating every data point is intractable, so we rely on sampling and statistical techniques.

• Lack of ground truth: For many prediction problems, there is no absolute source of truth to verify against. We can validate data for consistency and plausibility, but can‘t always prove correctness with 100% certainty.

• Biased data and feedback loops: ML models can inherit societal biases in the data used to train them. If we only validate that a model fits historical patterns, we may not catch unfair bias or discrimination. Careful model validation across sensitive attributes is needed for responsible AI.

• Changing environments: Data validation rules defined during training may not hold in production if the world changes. Rigid verification rules can reject too much data and cause models to break down. Dynamic validation techniques are an active area of research.

ML teams need a combination of automated data validation and human expert oversight to navigate these challenges effectively. Tools and statistical techniques should be leveraged to verify data quality and model behavior at scale, but human intervention will always be needed for the toughest judgment calls.

Conclusion

As machine learning becomes ubiquitous, we must hold ML models to the same rigorous standards as any other software system. Proactively validating data and verifying model behaviors throughout the ML lifecycle is crucial for building high quality, trustworthy, and ethical AI systems.

At each step of the ML workflow – from project planning, data exploration, model development, deployment, and maintenance – explicit data validation and verification steps are needed to ensure data and model integrity. By testing early and often, bad data and model issues can be caught before they cause real-world harm.

While ML verification is still an evolving field with active research challenges, ML teams can draw upon proven data testing principles and techniques to improve their model development practices. Combining automated data and model testing with human oversight – i.e. automating the routine checks so human experts can focus on overseeing the key decision points – is an effective approach.

As the AI historian David Spiegelhalter has said "AI that we can trust and control must be built on firm foundations." Data validation and model verification are those foundations for building robust, reliable, and trustworthy machine learning systems. No AI system should be deployed without them.