A Deep Dive into Apache YARN: Enabling Next-Generation Big Data Workloads

Apache Hadoop YARN (Yet Another Resource Negotiator) has been the de facto resource management platform for big data processing since its introduction in Hadoop 2.0. In this article, we‘ll explore YARN‘s history, architecture, components, and advanced capabilities through the lens of a AI/ML expert.

The Evolution of Hadoop Resource Management

To understand the significance of YARN, we need to go back to the early days of Hadoop. Hadoop was originally designed as a MapReduce system for batch processing of large datasets. The Hadoop 1.x architecture tightly coupled the MapReduce processing engine with the Hadoop Distributed File System (HDFS), with the JobTracker responsible for both cluster resource management and job scheduling/monitoring.

While this approach worked well for MapReduce, it made it difficult to run other types of distributed processing on Hadoop clusters. Each new engine had to implement its own resource management and job scheduling on top of HDFS. As the demand for real-time, iterative, and streaming processing grew, it became clear that Hadoop needed a more general-purpose resource management layer.

YARN, introduced in Hadoop 2.0, aimed to address this by decoupling cluster resource management from application lifecycle management. This would open up Hadoop to a wide variety of processing engines while allowing them to share cluster resources efficiently. YARN‘s architecture borrows concepts from distributed operating systems, treating a cluster as a collection of compute, memory, and I/O resources that can be dynamically allocated to applications.

YARN Architecture and Components

At its core, YARN provides APIs and services for cluster resource management, application lifecycle management, and security. Let‘s explore each of the key components in detail:

ResourceManager

The ResourceManager (RM) is the central authority that arbitrates resources among all applications in the system. It has two main components:

• Scheduler: Responsible for allocating resources to running applications according to constraints such as capacities, queues, and fairness. The Scheduler does not monitor or track application status – it relies on the ApplicationMaster for that. YARN supports pluggable scheduling policies via the Scheduler interface.

• ApplicationsManager: Provides APIs for clients to submit applications and request resources. It also maintains a list of running applications and handles ApplicationMaster failures.

The ResourceManager exposes APIs and UIs for cluster administrators to configure resource scheduling policies and monitor the cluster status.

Some key stats on typical ResourceManager deployments:

• Production Hadoop clusters often run thousands of nodes under a single RM
• The RM can handle 100K+ container allocation requests per second
• RM High Availability is used in over 80% of large deployments

NodeManager

The NodeManager (NM) is an agent that runs on each worker node and is responsible for launching containers, monitoring their resource usage, and reporting the same to the ResourceManager.

When an ApplicationMaster requests a set of containers, the Scheduler matches them to available resources and the corresponding NodeManagers are instructed to launch them. The NodeManager creates a dedicated working directory for each container, configures its environment, and executes the container process.

The NodeManager monitors each container‘s resource usage (CPU, memory, disk, network) and if it exceeds the allocation, preempts the container process. It also handles container failures and kills orphaned containers (e.g. from a failed ApplicationMaster).

In addition to running application containers, NodeManagers also provide auxiliary services such as log aggregation, container health monitoring, and cgroup-based resource isolation.

NodeManager advanced capabilities:

• Guaranteed and opportunistic CPU scheduling via CGroups
• Container launch time security and isolation via Docker/Appc containers
• Pluggable resource filtering and isolation mechanisms

ApplicationMaster

The ApplicationMaster (AM) is a framework-specific process that negotiates resources from the Scheduler, tracks the status and progress of the application‘s containers, and coordinates failover.

Each processing framework (e.g. MapReduce, Spark, Tez) implements its own AM to translate its programming model and semantics to the underlying YARN abstractions. For example:

• The MapReduce AM requests containers for map and reduce tasks, monitors their progress, and performs speculative execution
• The Spark AM requests containers for Spark executors, which in turn manage task execution
• The Tez AM builds a directed acyclic graph (DAG) for the user‘s job and dynamically resizes container allocations per vertex

Some common features of ApplicationMasters include:

• Matching container resource requests to cluster resources based on locality and other constraints
• Handling container failures by requesting new containers
• Coordinating efforts between containers running on different nodes
• Providing APIs and UIs for users to monitor the job status and logs

YARN ApplicationMaster implementations:

• Hadoop MapReduce: MRAppMaster
• Apache Spark: Spark YARN Client/AM
• Apache Tez: TezAM
• Microsoft Dryad: YARNDryadAppMaster

Container

The Container is YARN‘s basic unit of resource allocation. It is a logical bundle of resources (CPU, RAM, etc.) that is bound to a specific node and can be used by an application‘s process.

YARN‘s container model allows applications to specify their resource requirements in terms of multiple dimensions (memory, CPU, disk, network). The Scheduler matches these resource requests to the available resources on each node, taking into account priorities, capacities, and locality. This allows for efficient scheduling of heterogeneous workloads with varying resource needs.

Containers provide a certain degree of performance isolation by using Linux Cgroups to enforce limits on CPU, memory, disk I/O, and network usage. However, the level of isolation is not as strong as virtualization – all containers still share the same OS kernel on a node.

Container usage statistics:

• A large Hadoop cluster can have 100K+ running containers
• Containers sizes can range from <1 GB memory to 100s of GB
• Container launch latencies are in the 100s of milliseconds

Enabling Diverse Workloads

One of YARN‘s key value propositions is enabling diverse data processing workloads to run on a shared cluster. By decoupling resource management from application lifecycle management, YARN allows multiple processing frameworks to coexist on the same cluster.

This means that batch MapReduce jobs, real-time event processing, interactive SQL queries, machine learning jobs, and graph processing can all run on the same YARN cluster and efficiently share resources. Each processing framework implements its own ApplicationMaster to translate its semantics and resource needs to YARN abstractions.

Some popular processing frameworks that run on YARN include:

• Batch Processing: Hadoop MapReduce, Spark, Tez, Flink
• Machine Learning: Spark MLlib, TensorFlow-on-YARN, MXNet-on-YARN
• SQL and Data Warehousing: Hive, Impala, Presto, Drill
• Stream Processing: Spark Streaming, Storm, Samza, Flink
• Graph Processing: Giraph, GraphX, Tez

The rich ecosystem of YARN-enabled processing frameworks has made Hadoop clusters general-purpose big data platforms that can support a wide variety of use cases and workloads. This has greatly accelerated the adoption of Hadoop in the enterprise, with over 50% of the Fortune 500 running YARN-based Hadoop distributions.

Conclusion

Apache YARN has transformed Hadoop from a batch MapReduce platform to a multi-tenant, multi-workload data processing platform. By decoupling resource management from application lifecycle management, YARN has enabled a vibrant ecosystem of distributed processing frameworks to run efficiently on shared clusters.

YARN‘s architecture of a central ResourceManager, per-node NodeManagers, and per-application ApplicationMasters has proven scalable and flexible enough to power some of the largest big data clusters in the world. Advanced features like High Availability, Federation, Reservation, and Docker support have further extended YARN‘s appeal in the enterprise.

As an AI/ML expert, YARN‘s ability to natively support distributed training and model serving frameworks like TensorFlow and MXNet is particularly appealing. The ability to co-locate feature engineering, model training, and serving workloads on the same cluster allows for faster experimentation and more efficient resource utilization.

Looking ahead, there are many promising directions for YARN‘s evolution. Advanced use cases like GPU scheduling, cloud-native workloads, autoscaling, and long-running services are all active areas of innovation. By continuing to expand the types of workloads and resources it can manage, I believe YARN will solidify its place as the operating system for the modern big data platform.