Load data

As machine learning increasingly powers critical applications, it‘s more vital than ever to ensure our ML projects and experiments are reproducible. Reproducibility allows data scientists to collaborate effectively, iterate on experiments, and track results over time. It provides transparency and enables others to verify and build upon our work.

One powerful tool for enabling reproducible ML pipelines is YAML configuration files. YAML configs allow us to parameterize our code, separating the configuration from the logic. This makes it easy to tweak experiments without changing code.

In this post, we‘ll walk through an end-to-end example of building a reproducible ML pipeline using a YAML config file and Python code. We‘ll use the popular Kaggle Titanic dataset to predict passenger survival. Along the way, we‘ll discuss best practices for structuring projects and leveraging tools to maximize reproducibility.

What are YAML Config Files?

YAML (YAML Ain‘t Markup Language) is a human-readable data serialization format. It‘s often used for configuration files due to its simplicity and readability. A YAML file consists of key-value pairs and uses indentation to denote structure.

Here‘s a simple example of what a YAML config file for an ML project might look like:

data:
  train_path: data/train.csv 
  test_path:  data/test.csv
model:
type: RandomForestClassifier
params:
n_estimators: 100
max_depth: 5
random_state: 42
train:
cv_splits: 5
scoring: accuracy 

This config specifies the paths to the training and test data, the type of model to use (a Random Forest classifier) along with its hyperparameters, and settings for training like the number of cross-validation splits and the evaluation metric.

By separating these configurations from our code, we can easily experiment with different datasets, models, and hyperparameters without changing the underlying logic.

A Reproducible Titanic Survival Prediction Pipeline

Now let‘s see how to use a YAML config file in practice to build a reproducible ML pipeline. We‘ll use the Titanic dataset to predict passenger survival based on features like age, sex, passenger class, etc.

Here‘s what our project structure might look like:

titanic/
  ├── config.yaml
  ├── data/
  │   ├── train.csv
  │   └── test.csv 
  ├── pipeline.py
  ├── preprocess.py
  └── README.md

The key components are:

• config.yaml: Contains all our configuration parameters
• data/: Directory for storing our raw data
• pipeline.py: The main Python script that loads the config, runs the pipeline steps, and saves the model
• preprocess.py: A module containing data preprocessing functions
• README.md: Documentation for our project

Here‘s what our config.yaml file might look like:

data:
  train_path: data/train.csv
  test_path: data/test.csv
  target: Survived
preprocess:
impute_strategy: median
encode_strategy: onehot
model:
type: xgboost.XGBClassifier 
params:
n_estimators: 100
max_depth: 3
learning_rate: 0.1
subsample: 0.8
random_state: 42
train:
cv_splits: 5
scoring: roc_auc 
output:
model_path: model.pkl

This config includes the paths to our data, the name of the target variable, settings for imputation and encoding during preprocessing, hyperparameters for an XGBoost model, cross-validation settings for training, and the output path for the trained model.

Now in our pipeline.py script, we can load this config and use it to parameterize our pipeline:

import yaml
from preprocess import preprocess_data
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_auc_score 
import joblib
with open("config.yaml", "r") as f:
config = yaml.safe_load(f)

train_data = pd.read_csv(config["data"]["train_path"]) 
test_data = pd.read_csv(config["data"]["test_path"])
target = config["data"]["target"]

X_train, y_train, X_test, encoders = preprocess_data(train_data, test_data, target, 
impute=config["preprocess"]["impute_strategy"], 
encode=config["preprocess"]["encode_strategy"])

model = eval(config["model"]["type"])(**config["model"]["params"])

cv_scores = cross_val_score(model, X_train, y_train, 
cv=config["train"]["cv_splits"], 
scoring=config["train"]["scoring"])
print(f"Mean CV score: {cv_scores.mean():.3f}")

model.fit(X_train, y_train)

preds = model.predict_proba(X_test)[:, 1]
test_auc = roc_auc_score(y_test, preds)
print(f"Test ROC AUC: {test_auc:.3f}")

joblib.dump(model, config["output"]["model_path"])

This script loads the configuration settings, preprocesses the data, initializes and cross-validates the model, trains a final model on all training data, evaluates it on the test set, and finally saves the model to disk.

The preprocess_data function, defined in preprocess.py, handles missing value imputation and categorical encoding according to the strategies specified in the config file.

def preprocess_data(train_data, test_data, target, impute, encode):
    # Split features and target
    X_train, y_train = train_data.drop(target, axis=1), train_data[target]
    X_test, y_test = test_data.drop(target, axis=1), test_data[target]
# Impute missing values
if impute == "mean":
    imp = SimpleImputer(strategy="mean")
elif impute == "median":
    imp = SimpleImputer(strategy="median")
elif impute == "most_frequent":
    imp = SimpleImputer(strategy="most_frequent")
else:
    raise ValueError(f"Unknown imputation strategy: {impute}")

X_train = imp.fit_transform(X_train)
X_test = imp.transform(X_test)

# Encode categorical features  
if encode == "onehot":
    encoder = OneHotEncoder(handle_unknown="ignore")
elif encode == "ordinal":
    encoder = OrdinalEncoder()
else:
    raise ValueError(f"Unknown encoding strategy: {encode}")

X_train = encoder.fit_transform(X_train) 
X_test = encoder.transform(X_test)

return X_train, y_train, X_test, y_test, encoder
By parameterizing the preprocessing steps, we make it easy to experiment with different imputation and encoding strategies.
To run this pipeline, we simply execute:
python pipeline.py

The script will load the configurations from config.yaml, preprocess the data, train and evaluate the model, and save the final model to model.pkl. We can easily tweak the config file and re-run the pipeline to experiment with different settings.
Best Practices for Reproducible ML Pipelines
In addition to using YAML config files, there are several other best practices we can follow to ensure our ML pipelines are reproducible:


Separate configuration from code, as we did in the Titanic example. This makes it easy to experiment with different settings without modifying code.


Use relative paths in your config files and code. This ensures your project can be run on different machines without path issues. 


Version your data and models. Tools like DVC (Data Version Control) can help manage data and model versioning.


Document your experiments thoroughly. Use descriptive names for your config files (e.g. rf_experiment_1.yaml) and consider storing them in a dedicated experiments/ directory. Write clear READMEs detailing how to run your code.


Containerize your pipelines using Docker. This ensures your code will run consistently across different environments. You can specify your runtime environment (Python version, dependencies, etc.) in a Dockerfile.


Tools for Reproducible ML
There are several powerful tools and frameworks that can help us build reproducible ML pipelines with YAML configs:


Kubeflow Pipelines: An open-source platform for building and deploying portable, scalable ML workflows. Kubeflow Pipelines use YAML files to define the steps in a pipeline.


MLflow: An open-source platform to manage the ML lifecycle, including experimenting, reproducibility, and deployment. MLflow uses YAML files for its project configurations.


DVC: An open-source version control system for machine learning projects. DVC uses YAML files to define data pipelines and manage data versions.


These tools can help automate and standardize the process of building reproducible ML pipelines.
Additional Tips for Reproducibility
Here are a few more tips to keep in mind:

Use version control (Git) to track changes to your code and configurations.
Use virtual environments (venv, conda) to manage dependencies for your projects.
If you‘re not using Docker, consider using a tool like pipenv to manage your Python environment and pin your dependencies.
Document everything! In addition to your code, write clear READMEs, use descriptive variable and function names, and comment your code.

Conclusion
Reproducibility is critical for effective collaboration and iteration in machine learning. By using YAML configuration files and following best practices like separating config from code, versioning data and models, and containerizing pipelines, we can ensure our ML experiments are reproducible.
In this post, we walked through an example of building a reproducible pipeline for the Titanic dataset using a YAML config file. We covered best practices and tools for reproducibility.
I encourage you to adopt these practices in your own projects. Start by parameterizing your pipelines with YAML configs and work towards versioning your data and models and containerizing your code. Your future self (and collaborators) will thank you!
For more on reproducible ML, check out:

The Reproducible Research GitHub organization
This Towards Data Science post on reproducible machine learning
The Reproducible Research course on Coursera.

Happy coding and experimenting!
      
          
        How useful was this post?      
              
        Click on a star to rate it!      
        
      
                  
                      
                  
                      
                  
                      
                  
                      
                  
                      
              
    
    
          Submit Rating        
      Average rating 0 / 5. Vote count: 0    
    
      No votes so far! Be the first to rate this post.    
    
  
          


You May Like to Read,

Advancing Agriculture with Machine Learning: Crop Classification Techniques and Applications
Twitter-Based Gender Classification Using Machine Learning
A Complete Guide to Understanding Classification in Machine Learning
How Do Random Forests Really Work? An In-Depth Guide
Predicting Gender from Names using NLP and Python
Building ML Model to Distinguish If It's Human or ChatGPT
Classifying Emotions in Sentence Text Using Neural Networks
Streamline Your Machine Learning Workflow with Organized Pandas Dataframe Preprocessing