Getting Started with Apache Airflow: The Data Scientist‘s Guide

Introduction

Data is the lifeblood of modern organizations, and the volume and velocity of data are increasing exponentially. According to a report by IDC, the global datasphere is predicted to grow to 175 zettabytes by 2025 [1]. For data scientists and machine learning engineers, this presents both an opportunity and a challenge. The opportunity is to harness this data to drive insights and build intelligent applications. The challenge is to reliably and efficiently manage the complex data pipelines that make this possible.

This is where Apache Airflow comes in. Airflow is an open-source platform to programmatically author, schedule and monitor workflows. It has become the de facto standard for data orchestration, used by thousands of organizations worldwide including Airbnb, Twitter, PayPal, and Slack [2].

In this guide, we‘ll take a deep dive into Apache Airflow from a data scientist‘s perspective. We‘ll cover its core concepts, architecture, how to get started with it, and how it can be used to orchestrate machine learning pipelines. By the end, you‘ll have a solid foundation to start leveraging Airflow in your own data science and ML projects.

What is Apache Airflow?

Apache Airflow is an open-source workflow management platform. It started as an internal project at Airbnb in October 2014 before being open-sourced in June 2015. Airflow allows you to programmatically author, schedule and monitor workflows as Directed Acyclic Graphs (DAGs) of tasks.

A key principle of Airflow is that "configuration as code". The entire configuration of a workflow is defined in Python code, allowing for dynamic pipeline generation. This makes Airflow workflows versionable, testable, and maintainable.

Since its inception, Airflow has seen rapid adoption. As per a 2021 survey by Astronomer, Airflow is used by over 80% of data teams, with 61% of teams using it for both ETL and machine learning pipelines [3]. Airflow has also fostered a vibrant community, with over 1,900 contributors and 25,000 stars on GitHub as of 2023 [4].

Airflow‘s Architecture

To effectively use Airflow, it‘s important to understand its architecture and components. At a high level, Airflow consists of the following components:

• Web Server: This is the user interface for Airflow, it provides a control dashboard for users and maintainers to easily view and manage the state of their pipelines and tasks.

• Scheduler: The Scheduler is a process that handles both triggering scheduled workflows, and submitting Tasks to the executor to run.

• Metastore: This is the database where all metadata related to DAGs, tasks, variables, connections, etc. are stored. By default, Airflow uses a SQLite database, but in production, you would use a more robust database like PostgreSQL.

• Executor: An Executor is a process that executes tasks. There are several types of Executors, each suitable for different use cases and infrastructures. For example, the LocalExecutor runs tasks locally in parallel, while the CeleryExecutor distributes tasks to worker nodes.

• Worker: Workers are the processes that actually execute the logic of tasks. In the case of the CeleryExecutor, the workers pull tasks from a queue.

The diagram below illustrates how these components interact:

[Airflow Architecture Diagram]

When a DAG is triggered, either manually or via a schedule, the Scheduler creates a DagRun instance. The DagRun represents a specific execution of a DAG. The Scheduler then traverses the DAG‘s tasks in the correct order based on their dependencies, submitting them to the Executor. The Executor distributes these tasks to Workers for execution. As tasks get executed, their state (running, success, failed, etc.) is recorded in the Metastore. The Web Server reads this state information from the Metastore and displays it in the UI.

Defining a DAG

The core concept in Airflow is a DAG (Directed Acyclic Graph). A DAG is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.

Here‘s an example of a simple DAG that demonstrates several key concepts:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta

default_args = {
    ‘owner‘: ‘airflow‘,
    ‘depends_on_past‘: False,
    ‘start_date‘: datetime(2023, 1, 1),
    ‘email_on_failure‘: False,
    ‘email_on_retry‘: False,
    ‘retries‘: 1,
    ‘retry_delay‘: timedelta(minutes=5),
}

dag = DAG(
    ‘example_dag‘,
    default_args=default_args,
    description=‘A simple tutorial DAG‘,
    schedule_interval=timedelta(days=1),
)

def print_hello():
    return ‘Hello world from first Airflow DAG!‘

hello_operator = PythonOperator(
    task_id=‘hello_task‘,
    python_callable=print_hello,
    dag=dag,
)

bash_operator = BashOperator(
    task_id=‘bash_task‘,
    bash_command=‘echo Hello from Bash‘,
    dag=dag,
)

hello_operator >> bash_operator

Let‘s break this down:

• We first import the necessary libraries, including the DAG class and several Operators.
• We define a dictionary of default_args, which will be passed to the DAG. These include metadata like the owner, start date, retry settings, etc.
• We instantiate a DAG object, specifying its name, the default arguments, a description, and a schedule interval.
• We define a Python function print_hello that will be executed by a task.
• We create a PythonOperator task that will execute the print_hello function.
• We create a BashOperator task that will execute a Bash command.
• Finally, we define the dependencies between the tasks using the >> operator. This specifies that hello_operator must be executed before bash_operator.

When this DAG is run, Airflow will first execute the hello_task, and then the bash_task, respecting the dependency specified.

Orchestrating Machine Learning Pipelines with Airflow

One of the key use cases for Airflow is orchestrating machine learning pipelines. A typical ML pipeline might involve the following stages:

1. Data Ingestion: Pulling data from various sources like databases, APIs, or streaming platforms.
2. Data Preprocessing: Cleaning, transforming, and feature engineering on the raw data.
3. Model Training: Training a machine learning model on the prepared data.
4. Model Evaluation: Evaluating the trained model‘s performance on a test set.
5. Model Deployment: If the model performs well, deploying it to a production environment.

Each of these stages can be represented as a task or a set of tasks in an Airflow DAG. Airflow‘s various Operators and Sensors make it well-suited for orchestrating such pipelines. For example:

• The HttpSensor can be used to wait for data to be available in an API before triggering the data ingestion task.
• The PythonOperator can be used to execute data preprocessing and model training scripts.
• The BashOperator can be used to run shell commands for setting up environments or deploying models.
• The KubernetesPodOperator can be used to run tasks in a Kubernetes cluster, which is often used for distributed model training.

Here‘s a simplified example of what an ML pipeline DAG might look like:

from airflow import DAG
from airflow.contrib.sensors.file_sensor import FileSensor
from airflow.operators.python_operator import PythonOperator
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from datetime import datetime, timedelta

default_args = {...}

with DAG(‘ml_pipeline‘, default_args=default_args, schedule_interval=‘@daily‘, catchup=False) as dag:

    data_sensor = FileSensor(
        task_id=‘data_sensor‘,
        filepath=‘data/raw/{{ ds }}.csv‘,
        poke_interval=60,
        timeout=60*60,
        soft_fail=True
    )

    def preprocess_data(ds, **kwargs):
        # Data preprocessing logic here
        pass

    preprocess_task = PythonOperator(
        task_id=‘preprocess_data‘,
        python_callable=preprocess_data,
        provide_context=True
    )

    def train_model(ds, **kwargs):
        # Model training logic here
        pass

    train_task = PythonOperator(
        task_id=‘train_model‘,
        python_callable=train_model,
        provide_context=True
    )

    def deploy_model(ds, **kwargs):
        # Model deployment logic here
        pass

    deploy_task = PythonOperator(
        task_id=‘deploy_model‘,
        python_callable=deploy_model,
        provide_context=True
    )

    data_sensor >> preprocess_task >> train_task >> deploy_task

In this example:

1. The FileSensor waits for the raw data file to be available before triggering the preprocessing task.
2. The preprocessing task, implemented as a PythonOperator, preprocesses the raw data.
3. The training task, also a PythonOperator, trains the model on the preprocessed data.
4. Finally, the deploy task deploys the trained model.

The dependencies ensure that each task is only executed after the previous one has completed successfully.

Best Practices

When working with Airflow, there are several best practices to keep in mind:

1. Treat your DAGs as code: Your DAGs should be version controlled, tested, and reviewed just like any other code. Use a version control system like Git to manage your DAGs.

2. Keep your DAGs lean: Each DAG should have a single, well-defined purpose. If a DAG is getting too complex, consider breaking it up into smaller, more manageable DAGs.

3. Use Jinja templating: Airflow‘s built-in Jinja templating allows you to parameterize your DAGs, making them more dynamic and reusable.

4. Set up monitoring and alerting: Use Airflow‘s built-in tools and third-party integrations to monitor the health and performance of your DAGs. Set up alerts to notify you of any failures or anomalies.

5. Leverage Airflow‘s extensibility: Airflow has a rich ecosystem of plugins and extensions. Leverage these to extend Airflow‘s functionality and integrate it with your existing tools and platforms.

Case Studies

Many companies have successfully used Airflow to orchestrate their data and ML pipelines. Here are a few notable examples:

1. Airbnb: Airbnb, the birthplace of Airflow, uses it to manage their data pipelines which process and analyze petabytes of data every day [5].

2. Twitter: Twitter uses Airflow to orchestrate their ML pipelines, including model training, evaluation, and deployment. Airflow‘s extensibility allowed them to integrate it with their existing ML platform [6].

3. PayPal: PayPal uses Airflow to manage their data movement and ETL processes. They have over 1,000 DAGs running daily, processing over 4 petabytes of data [7].

4. Lyft: Lyft uses Airflow to power their data and ML platform, which includes ELT jobs, ML model training pipelines, and more. They have over 10,000 DAGs and process over 100 petabytes of data daily [8].

These case studies demonstrate the scalability and flexibility of Airflow in handling diverse data and ML workloads.

Alternatives to Airflow

While Airflow is a popular choice for workflow orchestration, it‘s not the only option. Some alternatives include:

1. Luigi: Developed by Spotify, Luigi is a Python package for building complex pipelines of batch jobs. Like Airflow, it handles dependency resolution, workflow management, and visualization.

2. Kubeflow: Kubeflow is an open-source platform for running ML workflows on Kubernetes. It‘s especially well-suited for ML workflows that involve distributed training and serving.

3. AWS Step Functions: AWS Step Functions is a serverless workflow orchestration service provided by Amazon Web Services. It integrates well with other AWS services and is suitable for less complex workflows.

The choice of tool depends on your specific requirements, existing tech stack, and team‘s skills.

Conclusion

Apache Airflow is a powerful and flexible platform for orchestrating data and machine learning workflows. Its principle of "configuration as code", rich set of operators and sensors, and extensive integrations make it well-suited for managing complex data pipelines.

In this guide, we‘ve covered Airflow‘s core concepts, architecture, and how to use it to orchestrate an ML pipeline. We‘ve also discussed best practices, real-world case studies, and alternatives.

As data becomes increasingly central to every business, tools like Airflow will only become more essential. Whether you‘re a data scientist, ML engineer, or data analyst, familiarity with Airflow is a valuable skill in today‘s data-driven world.

References

[1] IDC, "The Digitization of the World – From Edge to Core", 2018
[2] Airflow, "Airflow Users", https://airflow.apache.org/users.html
[3] Astronomer, "The State of Data Engineering", 2021
[4] GitHub, "Apache Airflow Repository", https://github.com/apache/airflow
[5] Airbnb Engineering & Data Science, "Airflow: A Workflow Management Platform", 2015
[6] Twitter Engineering, "Improving ML Workflow Orchestration with Airflow", 2021
[7] PayPal Engineering, "Airflow at Scale: A Financial Reporting Use Case", 2020
[8] Lyft Engineering, "Running Apache Airflow At Lyft", 2018