The Complete Guide to Redis Pub/Sub for AI & ML Applications

Introduction

Publish/subscribe (pub/sub) messaging is a key component of many real-time AI and machine learning systems. Pub/sub enables different services and modules to exchange data and events in a decoupled, scalable manner. Redis, the popular open-source data store, provides a simple yet powerful pub/sub implementation that is widely used in AI/ML stacks.

In this in-depth guide, we‘ll cover everything you need to know to effectively leverage Redis pub/sub in your AI/ML applications. We‘ll dive into:

• How Redis implements pub/sub
• The Redis pub/sub API and commands
• Scaling and performance considerations
• Pub/sub architecture patterns for AI/ML
• How to combine pub/sub with other Redis data structures for AI/ML use cases
• A real-world case study of Redis pub/sub in an AI application

Whether you‘re a data scientist, ML engineer, or architect, this guide will give you a solid foundation for using Redis pub/sub in intelligent, real-time applications.

How Redis Pub/Sub Works Under the Hood

Redis pub/sub builds on top of Redis‘ core data structures and networking layer. When a client issues a SUBSCRIBE or PSUBSCRIBE command, Redis stores the subscription information in two hash tables:

1. pubsub_channels: Maps channel names to a list of subscribed client IDs
2. pubsub_patterns: Maps pub/sub patterns to a list of subscribed client IDs

When a client publishes a message with PUBLISH, Redis looks up the channel in pubsub_channels and the message to each subscribed client. For pattern subscribers, Redis iterates through pubsub_patterns, checking the message channel against each pattern. For matching patterns, the message is sent to the subscribed clients.

Redis uses an efficient event loop to handle pub/sub messaging. The event loop continuously monitors for new client connections, incoming messages, and subscription changes. This allows Redis to handle a high throughput of pub/sub messages with low latency.

To track message delivery, Redis also maintains a pubsub_numpat counter of the total number of pattern subscriptions. This allows for quick lookup of whether any clients are subscribed to patterns without needing to iterate pubsub_patterns.

Under network partitions, Redis pub/sub uses a best-effort delivery approach. If a publisher and subscriber are on different sides of a network partition, the published message will not be delivered. Once the partition heals, publishers and subscribers will reconnect and resume message flow.

Redis Pub/Sub Performance & Scalability

Redis pub/sub is designed for high performance and scalability. In benchmark tests, a single Redis instance can handle:

• 1M subscriptions on a 4 vCPU / 14 GB RAM cloud instance (source)
• 10M subscriptions and 1.2M pub/sub messages/sec on a 96 vCPU bare metal machine (source)

Performance scales linearly with the number of Redis nodes, enabling very high pub/sub fan-out. Message latency is typically <1 ms at moderate load.

To scale Redis pub/sub beyond a single node, you can use:

• Redis Cluster: Messages published to each channel are processed by the master shard for that channel‘s hash slot. Subscribed clients must connect to the appropriate master shard.

• Sentinel: Clients can discover and connect to the current pub/sub master via Sentinel. The tradeoff is that Sentinel does not support cluster-style sharding.

• Independent masters: For maximum scalability, some large-scale Redis users shard pub/sub across a set of independent master nodes. Clients connect to a node based on a hash of the channel name. The tradeoff is that this requires custom sharding logic.

Pub/Sub for Scalable AI/ML Architectures

Pub/sub is a key pattern for scaling real-time AI/ML pipelines. The decoupled nature of pub/sub allows different stages of an AI/ML workflow to scale and evolve independently. Some architectural patterns we see frequently:

Event-driven inference

ML models often need to execute in real-time in response to events. For example, a ride-sharing app might want to predict demand in response to events like driver location updates, rider requests, weather changes, etc.

With Redis pub/sub, different services can publish these events to channels like driver-location-updates and ride-requests. The ML model service subscribes to these channels and runs the model in response to incoming messages. The model outputs can then be published back to result channels for other services to consume.

This pub/sub architecture allows the ML service to scale independently to handle the required throughput of model executions. It also allows new event publishers and result subscribers to be added without the ML service needing to be modified.

Real-time feature pipelines

Many ML models require computing real-time features from streaming data before making predictions. Redis pub/sub can be the connective tissue for these real-time feature pipelines.

Raw data events can be published to Redis channels by upstream producers. Feature computation services can subscribe to these raw data channels, process the data to generate features, and publish the computed features back to feature-specific channels. Model services then subscribe to the feature channels and use the features as input for generating predictions.

This modular pub/sub design makes it easier to add and update feature computations independently of the model service. It also provides the flexibility to share feature channels across multiple models.

Pub/sub for model ensemble orchestration

Advanced ML applications often use ensembles of multiple models to improve accuracy and handle different use cases. The outputs of one model may be used as an input to another model.

With Redis pub/sub, these model orchestration pipelines can be constructed as a series of pub/sub stages. Each model subscribes to the input channels it requires and publishes its outputs to channels for consumption by downstream models.

This allows the individual models to scale and update independently while maintaining a loosely coupled overall architecture. Adding or removing models from the ensemble is as simple as updating pub/sub channel routing.

Combining Pub/Sub with AI/ML-Centric Redis Features

While pub/sub alone is very useful for AI/ML applications, it becomes even more powerful when combined with other Redis features purpose-built for AI/ML use cases:

RedisAI

RedisAI is a Redis module that provides a wide range of capabilities for executing ML models and managing model lifecycle. With RedisAI, you can:

• Load DL/ML models from all major frameworks and execute models for both inference and training workloads
• Serve models from Redis so that online clusters can access shared models with Redis‘ sub-millisecond latency
• Build model inference DAGs that include both DL/ML models and traditional data processing operations

RedisAI fits naturally with Redis pub/sub. Published events can be sent to RedisAI for model inference, with the results stored back in Redis or published to result channels. RedisAI also supports running multiple model instances on a single server, enabling high throughput parallel processing of incoming pub/sub messages.

RedisGears

RedisGears is a serverless engine for Redis that enables data processing workflows to execute in response to events or on a schedule. RedisGears functions can be triggered by pub/sub messages, making it easy to build event-driven workflows.

For example, consider an e-commerce application that wants to train a product recommendation model based on real-time user clickstream data. With RedisGears, the application can:

1. Publish clickstream events to a Redis pub/sub channel
2. Register a RedisGears function to trigger on each event and write the event data to a Redis hash
3. Register a different RedisGears function to run hourly, read the clickstream data from Redis, train the model, and write the model back to Redis

RedisGears also supports Python libraries, enabling these workflows to use familiar ML libraries like NumPy, SciPy, and PyTorch. The combination of RedisGears and Redis pub/sub allows for complex real-time AI/ML pipelines to be built entirely in Redis.

Case Study: Real-Time Ad Bidding with Redis Pub/Sub

Let‘s walk through a concrete example of using Redis pub/sub to power a real-time AI application. In the real-time ad bidding space, online ad exchanges conduct auctions for ad impressions in milliseconds. Advertisers evaluate each impression opportunity and submit bids based on the expected value of showing an ad to that user.

Typically, ad bidders use ML models to predict the probability of a click or conversion event for each impression in order to calculate an optimal bid price. With Redis pub/sub, a real-time ad bidding system can work like this:

1. The ad exchange publishes each impression opportunity to a Redis pub/sub channel like impression-opportunities
2. Ad bidders subscribe to the impression-opportunities channel to receive opportunities in real-time
3. For each received opportunity, the bidder:
   • Extracts features like user ID, device, ad size, page URL, etc.
   • Looks up additional user features in Redis
   • Executes a bid prediction model (e.g. using RedisAI) to estimate probability of a click/conversion
   • Combines the predicted probabilities with ad campaign pricing and budget constraints to compute an optimal bid price
   • Publishes the bid price back to a Redis bids channel
4. The ad exchange subscribes to the bids channel and selects the winning bid for each impression

Redis enables this entire workflow to happen in milliseconds so that bids can be submitted before the ad exchange‘s deadline. Redis‘ ability to combine ultra-fast pub/sub with ML model serving (RedisAI) and low-latency user data storage makes it an ideal platform for latency-sensitive AI applications like real-time bidding.

Conclusion

Redis pub/sub is a versatile and essential tool for building scalable, real-time AI and ML applications. Its high performance, simple API, and ability to integrate with other Redis AI/ML features make it well-suited for a wide range of use cases from model inference to feature computation to result storage.

As the volume and velocity of data continue to grow and more applications require real-time intelligence, expect to see Redis pub/sub playing an increasingly critical role in the stack. Its unique ability to combine event-driven messaging with in-memory storage and built-for-purpose ML capabilities is a powerful accelerant for the real-time AI applications of the future.