The Ultimate Guide to Apache Oozie: Introduction and Getting Started

Apache Oozie is an open-source workflow scheduling and coordination system for managing big data processing jobs in Apache Hadoop. It allows you to define complex workflows as a collection of actions and specify their execution order while handling task dependencies and errors. Oozie aims to make it easier to plan, schedule and automate multi-stage Hadoop jobs involving tools like MapReduce, Pig, Hive and more.

In this comprehensive guide, we‘ll dive deep into Apache Oozie – exploring its core concepts, features, architecture, use cases and providing a hands-on tutorial to get you started. Whether you‘re completely new to Oozie or already have some experience, this article will equip you with a solid understanding of this powerful workflow management tool. Let‘s jump in!

What is Apache Oozie?

At its core, Apache Oozie is a scalable, reliable and extensible system for defining, scheduling and executing workflows of dependent Hadoop jobs. It provides a higher-level abstraction and orchestration layer on top of the Hadoop stack, allowing you to chain together map-reduce, Pig, Hive and other types of tasks into multi-step workflows.

Using Oozie, you define workflows using an XML-based Domain Specific Language (DSL) called the Oozie Workflow Definition Language. This declarative XML specifies the graph of actions that make up the workflow and the transitions between them. Oozie then takes care of scheduling the execution of actions, resolving their dependencies, triggering dependent workflows, handling failures and retrying tasks.

Some of the key features and benefits of Oozie include:

• Workflow management: Define workflows as a Directed Acyclic Graph (DAG) of actions. Specify their control flow and transition logic declaratively.

• Dependency management: Express dependencies between actions and ensure tasks execute in the correct order. Oozie handles resolving dependencies.

• Scheduling: Schedule workflows to run based on a certain frequency or based on data/time triggers using Oozie Coordinators.

• Error handling and retry: Configure error handling and retry behavior at the workflow or action level. Automatically retry failed tasks.

• Monitoring: Rich web UI and APIs for monitoring the status and progress of workflows and actions. Email alerts for job status.

• Integration: Deep integration with Hadoop and YARN. Extensible and pluggable to work with a variety of applications and tools.

• Scalability: Designed to scale to run thousands of concurrent workflows in a Hadoop cluster. Can manage long-running and complex jobs.

By providing these capabilities, Oozie greatly simplifies defining and managing complex data pipelines and ETL processes in Hadoop at scale. It promotes code reusability, modularity and eases the operational burden of running multi-stage jobs.

History and Evolution of Apache Oozie

Oozie was originally created at Yahoo in 2008-2009 to address the need for a scalable workflow scheduling and orchestration system for Hadoop. It started as a proprietary tool called "Hadoop Workflow" before being open sourced as an Apache Incubator project in 2011 and later graduating as a top-level Apache project in 2012.

Over the years, Oozie has continued to evolve and expand with new features and improvements. Some of the key milestones in its development include:

• Oozie v1.0 (2011): Initial release supporting workflows of map-reduce, Pig and file system actions. Basic coordinator functionality.

• Oozie v2.0 (2011): Added support for Hive and Sqoop actions. Bundle applications for packaging multiple coordinators.

• Oozie v3.0 (2012): Expanded workflow model with forks and joins. New powerful EL functions for parameterization.

• Oozie v3.1 (2012): Kerberos authentication support. HCatalog integration for data dependency tracking.

• Oozie v3.2 (2013): Improved performance and scalability optimizations. SLA monitoring capabilities.

• Oozie v4.0 (2014): Support for Hadoop 2.x and YARN. Pluggable action model for custom actions.

• Oozie v4.1-4.3 (2015-2017): Various bug fixes, security enhancements and minor feature additions.

• Oozie v5.0-5.2 (2018-2020): Support for Hadoop 3.x. Removal of deprecated streaming and pipes actions.

Today, Oozie is a mature and widely used workflow scheduler running in production at hundreds of organizations to power their big data processing pipelines. While development has slowed in recent years, it remains a critical and stable part of the Hadoop ecosystem.

Oozie Architecture and Components

Let‘s take a closer look at the main components that make up Apache Oozie and how they fit together:

1. Oozie Workflow Engine
   At the heart of Oozie is the workflow engine which is responsible for executing and managing workflows. It consists of the Workflow Engine and Coordinator Engine:

• Workflow Engine: Executes workflow jobs, determining which actions are eligible to run, submitting them to the cluster and tracking their status. Handles dependencies between actions.

• Coordinator Engine: Runs coordinator jobs, which are workflows that are initiated based on a schedule or data availability. Materializes workflows for each scheduled interval.

The workflow engine is a Java web-application that runs in a Java servlet container such as Apache Tomcat.

2. Oozie Client
   The Oozie client is a command-line utility and Java API for submitting and managing Oozie jobs. It communicates with the Oozie server using the Oozie REST API.

3. Oozie Workflow Definition
   Workflows are defined using the Oozie Workflow XML schema. The schema specifies the actions and control flow (start, end, decision, fork/join) that make up the workflow.

4. Oozie Coordinator Definition
   Oozie Coordinators are defined using the Oozie Coordinator XML schema. It specifies the datasets, inputs/outputs and schedule for running workflows.

5. Oozie Bundle
   An Oozie Bundle is a ZIP file that packages together all the workflows, coordinators, and other artifacts needed to run a job. It provides a convenient way to distribute and deploy Oozie applications.

6. ExtJS Web UI
   Oozie provides a web-based user interface implemented using the ExtJS JavaScript framework. It allows you to submit, monitor and manage Oozie jobs and view logs/output.

7. REST API
   Oozie exposes a REST API endpoint for all job and admin operations. This enables programmatic interaction and integration with Oozie from any HTTP client.

8. Database
   Oozie uses a relational database to store information about workflows, coordinators, actions and various job details. Supports databases like Derby, MySQL, Oracle, etc.

9. Hadoop Clients
   Oozie interacts with the broader Hadoop ecosystem using client libraries for HDFS, YARN, MapReduce, Pig, Hive and other supported tools.

All these components work together to enable defining, scheduling, executing and managing workflows in a Hadoop environment. The Oozie server handles processing workflow and coordinator definitions, stores state in the database, and interacts with the rest of the Hadoop stack to execute work.

Oozie Workflow Application

The core building block in Oozie is the workflow application, which is a collection of actions arranged in a control dependency DAG. The workflow definition consists of two main parts:

1. Workflow Definition
   An XML file that describes the workflow using the Oozie Workflow XML schema. It contains the details about the actions, their sequence and parameters.

2. Other Files
   The other supporting files such as JAR files, configuration files, scripts etc. that are needed by the actions in the workflow.

Here‘s a simple example of a workflow XML for a MapReduce workflow with a single action:

<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
    <start to="mr-node"/>
    <action name="mr-node">
        <map-reduce>
            <resource-manager>${resourceManager}</resource-manager>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/joe/out-dir"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.mapper.class</name>
                    <value>org.apache.oozie.example.SampleMapper</value>
                </property>
                <property>
                    <name>mapred.reducer.class</name>
                    <value>org.apache.oozie.example.SampleReducer</value>
                </property>
                <property>
                    <name>mapred.map.tasks</name>
                    <value>1</value>
                </property>
                <property>
                    <name>mapred.input.dir</name>
                    <value>/user/joe/in-dir</value>
                </property>
                <property>
                    <name>mapred.output.dir</name>
                    <value>/user/joe/out-dir</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

This workflow has a single MapReduce action node which runs a simple MapReduce job. The workflow starts at the start node, transitions to the mr-node action, and then either ends the workflow or transitions to the fail node if there is an error.

The MapReduce action specifies the details of the job such as the mapper/reducer classes, input/output paths and Hadoop configuration. It also has an optional prepare section to specify any setup steps before running the action.

Oozie workflows support a variety of control nodes and actions including:

• Control nodes: start, end, decision, fork, join
• Action nodes: map-reduce, pig, hive, hive2, sqoop, shell, spark, ssh, sub-workflow
• Other nodes: kill, global configuration

These building blocks provide a powerful way to define complex multi-step workflows involving different tools and processing stages.

Oozie Coordinators

While workflows define the execution logic for a job, coordinators allow you to schedule workflows to run on a recurring basis and launch them when data becomes available.

An Oozie coordinator application consists of a coordinator definition XML file specifying the datasets, inputs/outputs, and frequency for scheduling workflows. The workflows can use parametrization to access the dataset information.

Here‘s an example of a simple coordinator XML:

<coordinator-app name="sample-coord" frequency="${coord:days(1)}" start="2014-06-01T00:00Z" end="2014-12-31T00:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.4">
    <datasets>
        <dataset name="logs" frequency="${coord:days(1)}" initial-instance="2014-06-01T00:00Z" timezone="UTC">
            <uri-template>hdfs://localhost:9000/app/logs/${YEAR}${MONTH}/${DAY}</uri-template>
        </dataset>
    </datasets>
    <input-events>
        <data-in name="input" dataset="logs">
            <instance>${coord:current(0)}</instance>
        </data-in>
    </input-events>
    <action>
        <workflow>
            <app-path>hdfs://localhost:9000/app/sample-wf</app-path>
            <configuration>
                <property>
                    <name>resourceManager</name>
                    <value>localhost:8032</value>
                </property>
                <property>
                    <name>nameNode</name>
                    <value>hdfs://localhost:9000</value>
                </property>
                <property>
                    <name>queueName</name>
                    <value>default</value>
                </property>
                <property>
                    <name>inputDir</name>
                    <value>${coord:dataIn(‘input‘)}</value>
                </property>
                <property>
                    <name>outputDir</name>
                    <value>hdfs://localhost:9000/output/${coord:formatTime(coord:nominalTime(), ‘yyyy/MM/dd‘)}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>

This coordinator runs the referenced workflow application once a day when new data is available in the logs dataset. The workflow input and output paths are parameterized using the coordinator EL functions.

A coordinator will launch coordinator actions which will in turn trigger workflows for each dataset instance that matches the input dataset specification. Multiple coordinators can be further grouped using bundles to manage collections of related jobs.

Getting Started With Oozie

To get started using Oozie, you‘ll need a Hadoop cluster with Oozie installed and configured. Most Hadoop distributions include Oozie and have documentation on enabling it.

Here are the high-level steps to start using Oozie:

1. Install and configure Oozie on your Hadoop cluster following the distribution directions.

2. Define your Oozie workflows using the Workflow XML schema. You can write the XML directly or generate it via code in some cases.

3. Define your Oozie coordinators and bundles if you need scheduling of workflows.

4. Package your workflow/coordinator definitions along with any supporting files into an Oozie application.

5. Deploy the Oozie application to HDFS.

6. Submit and run the Oozie job using the Oozie client API or web UI. Monitor the progress and logs.

There are several ways to run Oozie jobs:

• Oozie command-line client: Use the oozie command to run jobs and perform other operations.

• Oozie Java client API: Use the Oozie client Java library to submit and manage Oozie jobs from your Java code.

• Oozie REST API: Interact with Oozie using its REST API endpoints, either directly or via an HTTP client.

• Oozie web UI: Use the Oozie web UI to submit jobs, monitor running jobs, and view logs and output.

The Oozie documentation provides detailed information on installation, configuration, schema reference, client APIs, and examples to help you get up and running.

Conclusion

Apache Oozie is a powerful workflow engine and scheduler for Hadoop that simplifies orchestrating complex data processing pipelines. With its XML-based DSL, extensible action types, scheduling capabilities, and integration with the broader Hadoop ecosystem, it provides a scalable and flexible platform for defining and managing workflows at scale.

In this guide, we covered the fundamentals of Oozie including its core concepts, architecture, workflow/coordinator model, and usage. While Oozie has its limitations and complexities, it remains a widely used and battle-tested solution for workflow management in Hadoop environments.

Oozie development has been relatively slow in recent years with Hadoop shifts to cloud environments and newer workflow tools like Apache Airflow gaining popularity. However, Oozie still powers many legacy Hadoop workloads and is likely to remain relevant in on-premise deployments.

I‘d encourage you to explore the Oozie documentation, try out the hands-on examples, and evaluate if it meets your workflow requirements. Feel free to post any questions or share your experiences in the comments below. Happy Oozying!