MLOps: Versioning Datasets with Git & DVC

In the world of software development, version control systems like Git have long been considered essential tools. They allow developers to track changes, collaborate smoothly, and maintain a clear history of their codebase. But what about in machine learning and data science? Can we apply the same versioning principles to our datasets and models?

The answer is a resounding yes – and it‘s becoming increasingly crucial as ML systems grow more complex and teams become more distributed. A 2021 survey by Algorithmia found that 87% of organizations have now adopted MLOps practices, with version control ranking as the #1 most important component [1].

In this post, we‘ll explore one powerful tool for bringing version control to ML datasets: DVC (Data Version Control). We‘ll dive into why dataset versioning is critical, walk through how to use DVC alongside Git, and highlight best practices and advanced workflows for maintaining reproducible, collaborative ML projects at scale.

The Need for Dataset Version Control

Machine learning models are fundamentally dependent on data. From the initial training data to new data encountered in production, the quality and composition of datasets can dramatically impact a model‘s performance. A 2021 study by IBM found that poor data quality costs organizations an average of $12.9 million per year [2].

Despite data‘s critical role, many organizations lack robust systems for versioning and tracking datasets as they evolve. Datasets are often scattered across different systems, named inconsistently, and updated haphazardly. This leads to a host of problems:

• Reproducibility issues: Without a clear record of which dataset was used to train a given model, it becomes difficult or impossible to reproduce results later.

• Collaboration friction: When multiple team members are making changes to datasets, it‘s easy to run into conflicts or accidental overwrites.

• Compliance risks: In regulated industries, being able to audit and track data lineage is crucial for compliance. Manual tracking is error-prone.

• Productivity drains: Time spent hunting down the right version of a dataset or debugging issues related to data changes is time not spent improving models.

According to a 2019 Kaggle survey of data scientists, dirty data is the most common barrier faced in ML projects, with 49% of respondents citing it as a challenge [3].

By applying rigorous version control to datasets, we can mitigate these issues and create a more stable, productive foundation for ML projects.

What is DVC?

DVC (Data Version Control) is an open-source version control system designed specifically for machine learning and data science projects. It builds on the core concepts of Git, but is optimized for handling large files and directories that are common in ML workflows.

With DVC, datasets and ML models are stored separately from code (either locally or in remote storage) but are referenced in a Git repository. This allows Git to efficiently handle code versioning, while DVC manages the versioning of data and model artifacts.

DVC was first released in 2017 and has quickly gained adoption in the ML community, with over 9,000 GitHub stars and 150 contributors [4].

Some of DVC‘s key features include:

• Data and model versioning: DVC uses a Git-like approach to version data and ML models. Each version is associated with a specific Git commit, allowing you to reproduce past results or roll back changes easily.

• Support for large files: Datasets in ML projects can often be gigabytes or even terabytes in size. DVC is designed to handle large files efficiently by storing them separately from Git and using file references.

• Remote storage integrations: DVC can push and pull data from remote storage services like S3, GCS, or Azure, making it easy to share datasets across a team or use in CI/CD pipelines.

• Data pipelines: DVC includes a built-in pipeline system for defining multi-stage data processing and modeling workflows. Pipelines are defined in a declarative YAML format and can be version controlled alongside datasets.

• Experiment tracking: DVC can log and track metrics from ML experiments, making it easy to compare results across different dataset versions, code changes, or hyperparameters.

• Data access control: DVC enables granular access control for datasets, ensuring that the right data is accessible to the right team members. Data access can be managed using Git permissions.

By combining these features, DVC provides a powerful framework for versioning and tracking all the inputs and outputs of an ML project in a unified way.

Versioning Datasets with DVC

Let‘s walk through a concrete example of using DVC to version datasets alongside code in a Git repository.

Step 1: Install DVC

First, make sure you have Git and Python installed. Then, install DVC using pip:

pip install dvc

Step 2: Initialize DVC

Next, navigate to your Git repository in the terminal and initialize DVC:

dvc init

This will create a .dvc directory for storing DVC‘s configuration and cache.

Step 3: Track a Dataset

Suppose we have a dataset file data.csv that we want to start tracking with DVC. We can add it to DVC using:

dvc add data.csv

DVC will move data.csv to its cache and create a small reference file data.csv.dvc. This reference file is what we‘ll commit to Git.

Step 4: Commit Changes

Now we can commit the DVC-tracked dataset to Git:

git add data.csv.dvc
git commit -m "Add initial dataset"

Notice that we‘re committing the .dvc reference file, not the actual data file.

Step 5: Push Data to Remote (Optional)

For team collaboration or CI/CD, we‘ll typically want to push our dataset to remote storage. With DVC, we can set up a remote using a command like:

dvc remote add -d myremote s3://mybucket/dvcstore

This sets up an S3 remote named "myremote". We can then push our data:

dvc push

The data is now available to be pulled by teammates or used in remote workflows.

Step 6: Make Changes and Track

As we make changes to our dataset over time, we can continue to track new versions with DVC:

dvc add data.csv
git add data.csv.dvc
git commit -m "Update dataset"
dvc push

DVC will efficiently store new versions of the data in its cache and allow us to switch between them using Git commits.

Reproducibility Benefits

One of the key advantages of using DVC for dataset versioning is the reproducibility it enables.

Suppose a teammate clones our Git repository and wants to reproduce an experiment we ran last week. With DVC, they can simply checkout the relevant Git commit and use dvc checkout to retrieve the exact dataset version used in that experiment – even if the dataset has changed in the meantime.

git checkout experiment-branch
dvc checkout

This ensures that everyone is working with a consistent version of the data, regardless of when they clone the repository or which version is currently in the main branch.

DVC‘s ability to create lightweight dataset snapshots and link them to specific Git commits is a powerful tool for maintaining reproducibility in complex ML projects.

Collaboration Workflows

Dataset versioning with DVC also streamlines collaboration within and across teams.

By pushing datasets to remote storage, teammates can access and use the same data without needing to manually transfer files. DVC‘s data access control features ensure that each collaborator has access to the right subsets of data for their work.

DVC‘s pipeline features are particularly useful for collaboration. By defining data processing and modeling steps in a DVC pipeline, we can ensure that each step is fully reproducible and share the pipeline with teammates. If a colleague needs to modify a step, they can do so in their own Git branch and propose the changes via a pull request.

This Git-like collaboration model, applied to datasets and pipelines, helps prevent overwrites and conflicts while maintaining a clear audit trail of changes.

Advanced Workflows

Beyond the basic versioning workflow, DVC supports a number of advanced use cases:

• Experiment Tracking: DVC can log arbitrary metrics during pipeline runs and associate them with the current Git commit. This allows us to compare metrics across dataset versions, code changes, and hyperparameters to find the optimal model. DVC provides commands like dvc metrics diff to compare experiments.

• Continuous Integration: DVC can be used in CI/CD pipelines to automatically test model performance when dataset changes are pushed. By including DVC metrics in the CI process, we can catch data-related regressions before they reach production.

• Data Governance: DVC‘s data access control and auditing features are useful for maintaining compliance in regulated industries. We can use DVC to maintain a record of who accessed which data when, and roll back any unauthorized changes.

• Partial Checkouts: For very large datasets, checking out the entire dataset for each experiment can be prohibitively slow. DVC supports partial checkouts, allowing us to retrieve only the subset of data needed for a given task.

By leveraging these advanced features, we can create more sophisticated, automated ML workflows while still maintaining full reproducibility and versioning.

Alternatives and Ecosystem

DVC is not the only tool in the MLOps versioning ecosystem. Other notable projects include:

• Pachyderm: An enterprise-scale data science platform that provides Git-like versioning for data, with a focus on scalability and Kubernetes integration.

• lakeFS: An open-source platform that provides Git-like operations for data lakes, with support for S3 and Azure Blob Storage.

• Dolt: A SQL database with Git-style versioning, designed for use in data science workflows.

Each of these tools has its own strengths and use cases, but DVC stands out for its close integration with Git, its support for a wide range of data storage backends, and its active open source community.

It‘s worth noting that DVC is also highly interoperable with other MLOps tools. It can be used alongside experiment tracking tools like MLflow, hyperparameter optimization tools like Optuna, and CI/CD platforms like Jenkins or GitHub Actions. This makes it a flexible choice for teams with existing ML infrastructure.

Conclusion

As machine learning matures and moves from experimental projects to production systems, robust versioning practices for datasets become increasingly critical. Without versioning, ML teams risk losing reproducibility, introducing data-related bugs, and spending hours on manual data management.

DVC provides a powerful, Git-native solution for versioning and tracking datasets and models. By making data a first-class citizen in version control, DVC brings the same reliability and collaboration benefits to data that we‘ve long enjoyed for code.

In this post, we‘ve explored the key features of DVC, walked through a basic workflow, and discussed some of the advanced capabilities it enables. We‘ve also touched on the broader ecosystem of tools for MLOps versioning and how DVC fits in.

Whether you‘re just starting out with ML or managing a complex production system, adopting DVC can help streamline your workflow, improve reproducibility, and unlock new opportunities for collaboration. By versioning your data with the same rigor as your code, you‘ll be well on your way to building more reliable, scalable ML systems.

To dive deeper into DVC, check out the official documentation and GitHub repository. You can also join the DVC community to learn from other practitioners and contribute to the project.

Happy versioning!