A Comprehensive Guide to Building Machine Learning Pipelines with Kedro

Machine learning and data science projects can quickly become complex and difficult to maintain as the codebase grows. Without proper organization and engineering best practices, projects turn into a tangled mess of spaghetti code, making it hard to collaborate, reproduce results, and deploy to production. Fortunately, open-source workflow tools like Kedro aim to solve these challenges by providing a standardized framework for building modular, maintainable, and production-ready pipelines.

In this in-depth guide, we‘ll walk through using the Kedro framework to build a complete machine learning pipeline for a news article classification task. Along the way, we‘ll explore Kedro‘s core concepts, see how it facilitates collaboration and reproducibility, and share best practices for deploying Kedro projects. Let‘s dive in!

What is Kedro?

Kedro is an open-source Python framework that applies software engineering best practices to machine learning code. It prescribes a standard project template and a structured way to organize code into modular pipelines. Kedro was originally developed by QuantumBlack, a McKinsey company, and is now maintained by an open-source community.

Some key features and benefits of Kedro:

• Seperates configuration from code with a centralized ‘conf‘ folder
• Enforces a modular architecture using nodes (Python functions) and pipelines (DAGs of nodes)
• Manages project data effectively using the Data Catalog and versioning
• Makes projects reproducible with a standard project template
• Facilitates collaborations with strong conventions and a system for configuration environments
• Integrates with many plugins enabling packaging, visualization, and deployment
• Supports incremental computation and caching out-of-the-box

With Kedro, data scientists can focus on solving business problems, while having confidence their work is reproducible and can be seamlessly put into production when ready. Let‘s now understand some of Kedro‘s core concepts before diving into building our classification pipeline.

Understanding Kedro‘s Core Concepts

Kedro is built around three main concepts – the data catalog, nodes, and pipelines. Understanding these key abstractions is crucial to effectively leverage the framework.

The Data Catalog

The data catalog is the single source of truth for project data. It maps dataset names to their locations, formats, and any load arguments needed. Kedro uses the catalog to resolve dependencies between tasks and ensure a pipeline is reproducible.

Here‘s an example data catalog in YAML format:

news_data_raw:
  type: pandas.CSVDataSet
  filepath: data/01_raw/news_data.csv

news_data_processed:
  type: pandas.ParquetDataSet  
  filepath: data/02_intermediate/news_data_cleaned.pq

This catalogs two datasets – a raw CSV and cleaned Parquet file. With this declarative configuration, Kedro knows exactly what data a pipeline needs and produces. Anytime we reference a dataset in code, Kedro looks it up in the catalog.

Nodes

Nodes are the building blocks of pipelines in Kedro. They are simply Python functions that encapsulate a unit of work, like loading data, feature engineering, or model training. What makes them special is they explicitly declare their inputs and outputs, which Kedro uses to wire nodes together.

Here‘s an example node definition:

def preprocess_data(news_data_raw: pd.DataFrame) -> pd.DataFrame:
    clean_data = news_data_raw.dropna()
    clean_data["content"] = clean_data.title + " - " + clean_data.description 
    return clean_data

This node takes in the raw DataFrame, cleans it, and returns the processed DataFrame. Notice the input and output type hints – these provide Kedro the information it needs to build a pipeline. Speaking of pipelines, let‘s see how nodes come together to create them.

Pipelines

A pipeline organizes nodes into a directed acyclic graph (DAG) based on their input and output relationships. Kedro automatically resolves the execution order using the dependencies declared between nodes. A pipeline can be made up of sub-pipelines, allowing for modularity and reuse.

Here‘s a simple pipeline definition using the previous node:

from kedro.pipeline import Pipeline, node

data_processing_pipeline = Pipeline(
    [
        node(
            func=preprocess_data,
            inputs="news_data_raw",
            outputs="news_data_processed",
            name="preprocessing_node",
        )
    ]
)

This pipeline contains a single node that processes the raw data. The node‘s inputs and outputs refer to the keys in our data catalog. When we run this pipeline, Kedro will ensure the catalog datasets exist, execute the node, and save its output back to the catalog.

With the core concepts covered, we‘re ready to get our hands dirty building a real Kedro pipeline!

Building a News Classification Pipeline

We‘ll now walk through creating a Kedro project to classify news articles into categories like sports, technology, and business. The pipeline will include loading data, cleaning and feature engineering, training a classifier, and evaluating the model. We‘ll be using the popular AG News dataset.

Step 1 – Create a New Project

First, make sure you have Kedro installed:

pip install "kedro[pandas.CSVDataSet, pandas.ParquetDataSet]"

Next, create a new project:

kedro new

Select the default project template and name the project "news_classifier". Enter the created directory and download the AG News dataset into data/01_raw.

Step 2 – Define the Data Catalog

Let‘s declare the project‘s input data in conf/base/catalog.yml:

news_data_raw:
  type: pandas.CSVDataSet
  filepath: data/01_raw/news_data.csv

label_encoder:
  type: pickle.PickleDataSet
  filepath: data/02_intermediate/label_encoder.pkl  

news_data_cleaned:
  type: pandas.ParquetDataSet
  filepath: data/03_primary/news_data_cleaned.pq

news_classifier:
  type: pickle.PickleDataSet
  filepath: data/06_models/news_classifier.pkl

model_metrics:
  type: json.JSONDataSet
  filepath: data/08_reporting/model_metrics.json

These declarative dataset definitions tell Kedro where to find the input data and where to save intermediate and output artifacts.

Step 3 – Create the Processing Pipeline

Our first pipeline will load and preprocess the news data. Use Kedro‘s CLI to create a new pipeline:

kedro pipeline create data_processing

This generates a pipeline structure under src/news_classifier/pipelines/data_processing. Let‘s add a few processing nodes in data_processing/nodes.py:

import pandas as pd
from sklearn.preprocessing import LabelEncoder

def preprocess_news_data(news_data_raw: pd.DataFrame) -> pd.DataFrame:
    news_data_raw["content"] = news_data_raw.title + " - " + news_data_raw.description
    news_data_cleaned = news_data_raw.loc[:, ["content", "class_index"]]
    return news_data_cleaned

def encode_labels(news_data_cleaned: pd.DataFrame) -> (pd.DataFrame, LabelEncoder):
    label_encoder = LabelEncoder()
    encoded_labels = label_encoder.fit_transform(news_data_cleaned["class_index"]) 
    news_data_cleaned["label"] = encoded_labels
    return news_data_cleaned, label_encoder

The first node combines the title and description fields. The second fits a LabelEncoder to convert string labels to integers.

Now let‘s hook up the nodes in data_processing/pipeline.py:

from kedro.pipeline import Pipeline, node
from .nodes import preprocess_news_data, encode_labels

def create_pipeline(**kwargs):
    return Pipeline(
        [
            node(
                func=preprocess_news_data,
                inputs="news_data_raw",
                outputs="news_data_cleaned",
                name="preprocess_node",
            ),
            node(  
                func=encode_labels,
                inputs="news_data_cleaned",
                outputs=["news_data_cleaned", "label_encoder"],
                name="label_encoding_node",  
            )
        ]
    )

Great, we‘ve created our first pipeline! Let‘s make sure it runs:

kedro run --pipeline=data_processing

You should see the processed data saved under data/03_primary.

Step 4 – Create the Modeling Pipeline

On to the fun part – training a classifier! Create another pipeline for modeling:

kedro pipeline create data_science  

Add a modeling node under data_science/nodes.py:

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression  
from sklearn.metrics import classification_report

def train_classifier(news_data_cleaned: pd.DataFrame) -> (LogisticRegression, dict):
    X = news_data_cleaned.content
    y = news_data_cleaned.label

    X_train, X_test, y_train, y_test = train_test_split(X, y)

    vectorizer = TfidfVectorizer()
    X_train_vect = vectorizer.fit_transform(X_train)
    X_test_vect = vectorizer.transform(X_test)

    classifier = LogisticRegression()  
    classifier.fit(X_train_vect, y_train)

    y_pred = classifier.predict(X_test_vect)
    metrics = classification_report(y_test, y_pred, output_dict=True)

    return classifier, metrics

This node splits the data, fits a TF-IDF vectorizer and logistic regression model, and returns the trained classifier and evaluation metrics.

Wire it up in data_science/pipeline.py:

from kedro.pipeline import Pipeline, node  
from .nodes import train_classifier

def create_pipeline(**kwargs):
    return Pipeline(
        [
            node(
                func=train_classifier,
                inputs="news_data_cleaned",  
                outputs=["news_classifier", "model_metrics"],
                name="train_classifier_node",
            )
        ]
    )

Let‘s run the pipeline and check the results:

kedro run --pipeline=data_science

Navigate to data/08_reporting/model_metrics.json to view the model‘s performance. Not too shabby for our first go!

Visualizing the Pipeline

As pipelines grow more complex, it‘s useful to visualize them to understand the flow of data. Kedro comes with a great built-in visualization tool. Install Kedro-Viz and launch the UI:

pip install kedro-viz
kedro viz

Here‘s what our classification pipeline looks like:

The UI shows each dataset and node, and clearly presents the relationships and execution order between them. This view makes it easy to communicate the high-level workflow to others.

Deploying to Production

We‘ve built a working classifier pipeline, but how do we deploy it to production? Kedro integrates with many orchestration tools to schedule pipelines. A common approach is using Kedro-Airflow to convert your Kedro pipeline into an Airflow DAG.

First install the plugin:

pip install kedro-airflow

Running kedro airflow create will generate an Airflow DAG file from your Kedro pipeline. Set the appropriate schedule and deploy your Airflow instance to run pipelines remotely on a schedule.

Other deployment options include:

• Kedro-Docker for packaging pipelines in containers
• Kedro-Kubeflow for running on a Kubernetes cluster
• Kedro-AWS-Batch for execution on AWS Batch

The key is Kedro standardizes project structure, so you can pick your preferred execution engine and deployment target.

Conclusion

To recap, we demonstrated how to:

• Understand Kedro‘s key concepts – the data catalog, nodes, and pipelines
• Build a news classification pipeline with data processing and modeling steps
• Visualize Kedro pipelines
• Deploy Kedro workflows with Apache Airflow

Kedro offers data scientists a productive framework for developing modular, maintainable, and deployable pipelines. It seamlessly integrates software engineering best practices into machine learning workflows. Key benefits include:

• Improved project structure and reproducibility
• Abstraction of configuration from code
• Flexibility to choose pipeline components
• Integration with many execution engines
• Support for incremental builds and caching
• Pipeline parameterization and versioning
• Smooth transition from prototype to production

Having used Kedro for real-world projects, I‘ve found it strikes a nice balance between flexibility and standardization. The structured workflow keeps projects organized and enables effective collaboration. At the same time, you retain full control over pipeline logic by writing standard Python.

While Kedro has a bit of a learning curve, it‘s well worth the investment to create production-grade ML pipelines. It serves as a strong foundation for MLOps, seamlessly integrating with other tools in the ecosystem. Give it a shot on your next project!