Apache Flume: The Flexible and Reliable Big Data Ingestion Tool

In today‘s fast-paced digital world, organizations are collecting massive volumes of data from a wide variety of sources – application logs, clickstream data, IoT sensors, social media feeds and more. According to IBM, 90% of the data in the world today has been created in the last two years alone, at 2.5 quintillion bytes of data a day!

To gain insights and make data-driven decisions, companies need a way to efficiently and reliably ingest this fast-moving big data into platforms like Hadoop or cloud object storage for long-term persistence as well as stream processing engines for real-time analysis. This is where Apache Flume comes in.

What is Apache Flume?

Apache Flume is an open-source, distributed system for collecting, aggregating and transporting large amounts of streaming event data. Originally developed by Cloudera and now a top-level Apache project, Flume is designed to be a highly reliable, configurable and scalable tool that can handle massive data volume in real-time.

At its core, Flume is built around a simple concept of data flows that consist of three main components:

Sources that receive data from external sources and convert it into Flume events
Channels that buffer events until they are consumed by sinks
Sinks that remove events from channels and put them into external repositories like HDFS, HBase or forward them to other Flume agents

Using this architecture, Flume can create complex, multi-hop flows where events travel from source to sink, potentially passing through multiple Flume agents along the way for aggregation, transformation and routing.

Flume Architecture Deep Dive

Let‘s take a closer look under the hood at some of the key architectural components of Flume.

Flume Events

The basic unit of data that is transported by Flume is called an event. A Flume event consists of a byte payload and optional string headers used to store metadata or routing information. Events are immutable and have a unique ID associated with them.

Flume Agents

A Flume agent is a JVM process that hosts the components through which events flow from an external source to the next destination.

A Flume agent consists of:

Single Source that receives events from external source
One or more Channels that buffer events until processed
One or more Sinks that remove events from channel and forward to next agent or store in external repository
Channel Selectors determine which channels to deposit events into based on pre-defined criteria
Sink Processors choose which sink to forward events to in case of multiple sinks

The source and sinks within the given agent run asynchronously with the events staged in the channel. This allows sources to operate at a different rate than sinks and decouple input rate from output rate.

Reliability

An important aspect of Flume is its support for reliable event delivery between agents in a flow. When a source sends an event to a channel, it commits a transaction. Similarly, when a sink removes an event from a channel, it also commits a transaction. This ensures that an event is delivered exactly once from point to point within a given flow.

In the case of multi-hop flows, the sink from one Flume agent and source from the next are chained together to ensure reliable delivery across the entire pipeline.

Flume Configuration

Flume agents are configured using a properties file that specifies which sources, channels and sinks to use as well as their properties. Here is an example configuration file that sets up an agent to collect log data and store it to HDFS:

# Named components
agent.sources = webserver-log-source
agent.channels = hdfs-channel 
agent.sinks = hdfs-sink

# Source
agent.sources.webserver-log-source.type = exec
agent.sources.webserver-log-source.command = tail -F /var/log/apache2/access.log

# Channel
agent.channels.hdfs-channel.type = memory
agent.channels.hdfs-channel.capacity = 1000

# Sink
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://namenode/flume/webserver/%y-%m-%d
agent.sinks.hdfs-sink.hdfs.fileType = DataStream

# Bind components
agent.sources.webserver-log-source.channels = hdfs-channel
agent.sinks.hdfs-sink.channel = hdfs-channel

This configuration tails an Apache web server access log and sends each event through a memory channel to HDFS where the events are stored in files partitioned by date. Flume has many out of the box options for sources, channels and sinks as well as the ability to create custom implementations.

Flume Usage Statistics

Apache Flume has seen significant adoption in the industry since its initial release in 2011. Some statistics on its usage:

Over 80% of Cloudera customers use Flume to ingest data into their Hadoop clusters (source)
Flume can scale to handle 100,000+ events per second on modest hardware (source)
At its peak, Pinterest used Flume to ingest over 100 terabytes per day into their Hadoop data warehouse (source)
Apache Flume was among the top 5 most popular Apache projects by commits in 2019 (source)

Flume and Machine Learning

One of the key drivers of Flume adoption has been the rise of Hadoop-based data lakes that serve as the foundation for machine learning and AI applications. By collecting and channeling raw event data into HDFS, Flume helps data scientists access the massive training datasets they need to build complex models.

Some examples of machine learning powered by Flume-delivered data:

Anomaly Detection – PayPal uses Flume to ingest web traffic and machine logs into their Hadoop data lake. This data is used to train unsupervised learning models that detect fraud and cyber threats in real-time. (source)
Recommendation Engines – Ecommerce companies like Overstock.com use Flume to capture granular user interaction data that trains machine learning models to predict which products customers are most likely to purchase. (source)
Predictive Maintenance – GE uses Flume to collect sensor data from their fleet of jet engines, power turbines and other industrial assets. This IoT data feeds neural networks that predict when equipment failures will occur so that maintenance can be performed proactively. (source)

According to a report by Markets and Markets, the global big data market is expected to grow from $138.9B in 2020 to $229.4B by 2025. A large portion of that data will be event streams generated by digital applications and devices. As the demand for ML applications that process this streaming data continues to accelerate, Apache Flume will play an increasingly important role in building the next generation of intelligent systems.

Flume Roadmap and Future Development

As an open source project, Apache Flume is under active development by a community of contributors. Some key items on the roadmap include:

Kafka Integration – Enhancements to the Kafka source and sink to support new features in Kafka 2.x such as transactions and header support (FLUME-3349)
Containerization – Support for running Flume agents as Docker containers and better integration with container orchestration platforms like Kubernetes (FLUME-3179)
SQL Sinks – New sinks for real-time ingestion into SQL stores like MemSQL and Kudu to support live dashboarding and interactive analytics use cases (FLUME-2977)

Looking further into the future, there are opportunities to integrate Flume with additional streaming platforms such as Apache Pulsar, Apache NiFi and Apache Flink. There is also potential for Flume to provide built-in stream processing capabilities such as event filtering, transformation and aggregation today requiring separate tools.

"Flume has always been a versatile and rock-solid solution for building scalable streaming pipelines," says Arvind Prabhakar, CTO of StreamSets and Flume committer. "The Flume community is responding to new customer use cases and the evolution of the big data ecosystem. The future is bright for Flume."

Conclusion

Apache Flume is a battle-tested and versatile open-source tool for large-scale log collection and streaming to big data stores. By providing a scalable, fault-tolerant and configurable architecture for buffering and routing events, Flume powers real-time ingestion pipelines at petabyte scale. As big data and streaming become the foundation for machine learning, Flume will continue to serve as critical infrastructure to build the next wave of intelligent applications.

Apache Flume: The Flexible and Reliable Big Data Ingestion Tool

What is Apache Flume?