# Scaling Slack‘s Data Architecture to Power Collaboration and AI/ML Innovation

- Canonical: https://33rdsquare.com/slack-data-engineering-design-and-architecture/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

## Introduction

Since its launch in 2013, Slack has become a transformative force in enterprise communication and collaboration. From startups to Fortune 100s, millions of professionals rely on Slack to stay connected and aligned in an increasingly digital workplace.

Slack‘s growth has been nothing short of explosive – the company now boasts over 12 million daily active users across more than 600,000 organizations [1]. Supporting this scale while delivering a fast, reliable user experience presents significant data architecture challenges:

- Rapidly ingesting and indexing massive volumes of messages, files and events
- Storing and serving datasets that grow by terabytes a day
- Enabling near real-time data synchronization between clients and backend stores
- Processing complex queries and push notifications within milliseconds
- Training and productionizing machine learning models that continuously improve the user experience

In this post, we‘ll take a deep dive into how Slack re-architected its data platform to overcome these challenges and evolve from a single-tenant web app to the distributed, AI-powered collaboration engine it is today. We‘ll examine the key components of Slack‘s modern data architecture, share best practices and principles, and discuss how this foundation unlocks exciting potential through the application of AI/ML.

## The Journey to Lazy Loading

In Slack‘s original architecture, clients would download a complete snapshot of all the data required to render the UI whenever a user logged in or refreshed. For smaller workspaces this worked well – data could be downloaded quickly and having it available locally enabled a snappy user experience.

However, as the size of workspaces grew, so too did the size of the snapshots. For large enterprises, snapshots could reach _tens to hundreds of megabytes_ [2]. Loading this much data upfront led to unacceptable increases in connection times and memory usage, especially on mobile.

To address this, Slack moved to a "lazy loading" architecture. Clients now receive a minimal snapshot to render the initial UI, then fetch additional data on-demand as the user navigates around. Enabling lazy loading required introducing a new service, Flannel, to handle the fetching and caching of this data.

### Enter Flannel

Flannel is a geo-distributed query engine and cache that serves as the data access layer for Slack clients [3]. It‘s built to handle a huge volume of queries with predictable low latency. Some key aspects of its design:

- Stateless, shared-nothing architecture comprised of many small instances that can be scaled independently
- In-memory datasets organized by workspace that can serve most queries without touching disk
- Automatic dataset management – instances load workspace data on first client request and evict it when no longer needed
- Complex query support, including secondary indexes to efficiently filter and aggregate data
- Write-through caching – instance forwards mutations to the primary datastore to avoid inconsistency

With Flannel in place, snapshots decreased in size by orders of magnitude. P99 connection times for large enterprises dropped from 20+ seconds to under 5 seconds [2], while backend costs decreased due to more efficient utilization.

### Just-in-time Annotations

Beyond serving queries directly, Flannel also _proactively_ pushes data to clients in a technique called "just-in-time annotations". As backend events are streamed to clients, Flannel predicts which associated records will be needed to display the event and includes them in the response payload.

For example, when a message is sent, Flannel checks if the sender‘s user record is already in the client‘s cache. If not, it will _annotate_ the message payload with the full user object, saving a followup client request.

This technique, combined with a special _client-side_ cache in the Electron desktop app, improved latency by _3-4x_ for common operations like switching channels and displaying user profiles [4].

## Adopting Pub/Sub for Scale and Flexibility

The initial lazy loading implementation had two key inefficiencies:

1. Flannel instances had to process duplicate events for each client subscribed to a channel
2. The in-memory dataset for a workspace couldn‘t be loaded until the first client connected, leading to higher p99 latencies

To solve this, Slack re-architected event delivery to use a pub/sub model. With pub/sub:

- Backend services publish events to a message broker with _at-least-once_ delivery semantics
- Flannel instances subscribe to the broker and receive events exactly once
- Subscriptions are sharded across Flannel instances by workspace to evenly distribute load
- Flannel can pre-warm in-memory datasets before clients connect to optimize p99 latency

Slack built a custom pub/sub system called _Disq_ that‘s optimized for their use case and deployments [5]. Some key benefits:

- Easily handles _10M+ msg/sec_ at peak across all workspaces
- Built-in _server-side filtering_ to only deliver relevant events to each Flannel instance
- _Horizontally scalable_ – can add more brokers and Flannel instances to handle growth
- Supports _multi-region replication_ for fault tolerance and disaster recovery

Switching to pub/sub yielded huge efficiency gains for Slack. The number of events Flannel needed to process per workspace _decreased by 99.8%_ (500K → 1K msg per 10s). The p50 and p99 latency of common queries _improved by over 50%_ [6].

## Enabling Deeper Insights with Change Data Capture

As Slack increasingly became a _system of record_, they saw opportunities to provide rich insights by combining data across all the tools a company uses. To make this possible, Flannel added support for _change data capture (CDC)_ [7].

With CDC, every mutation to a workspace‘s data is captured and streamed in real-time to a central repository. This enables Slack to:

- _Analyze_ data access patterns to optimize performance and cost
- _Aggregate_ data with other sources to power advanced insights and machine learning
- _Archive_ data for auditing, compliance and governance purposes
- _Index_ data in downstream systems for fast, flexible querying

CDC is powered by a custom _event sourcing_ framework built on top of MySQL [8]. Whenever entity is created, updated or deleted, the framework:

1. Writes the mutation to the database table normally
2. Writes a corresponding event to an append-only event log table
3. Publishes the event to Disq for consumption by downstream subscribers

This dual-write approach ensures events are never lost, even if the subscriber is offline. It also enables replaying events to reconstruct the state of an entity at any point in time.

## Leveraging AI/ML to Optimize Performance and User Experience

With a scalable, flexible data architecture in place, Slack is now able to leverage AI and machine learning in powerful ways across the product. Some examples:

- **Search & Discovery**: _Natural language understanding_ models process all messages to extract key terms, entities and intent. These signals are used to power contextual search, recommendations and intelligent results filtering, making it easier for users find the information they need.
- **Anomaly Detection**: _Unsupervised learning_ algorithms analyze data access patterns and system metrics to automatically detect and alert on anomalies. This helps engineers proactively identify and fix issues before they impact users.
- **Channel Insights**: _Text classification_ and _sentiment analysis_ models are used to auto-generate channel insights. For example, Slack can detect common questions being asked in a channel and suggest adding them to the channel topic or pinning relevant resources.
- **Smart Notifications**: _Activity classification_ models are used to suppress notifications for low-value messages and prioritize ones that require a user‘s attention. Over time, the models learn the user‘s individual preferences to personalize the experience.
- **Predictive Data Management**: _Usage forecasting_ models analyze data growth and access patterns to predict future storage and caching requirements. This enables Slack to proactively scale infrastructure and optimize data placement to ensure a consistent user experience.

To support these AI/ML applications, Slack has built out a comprehensive _feature platform_ that enables machine learning teams to rapidly build and deploy models [9]. The platform provides:

- Managed _feature stores_ for consistent, point-in-time access to training data
- Serverless _ETL pipelines_ to process and aggregate data from disparate sources
- Autoscaling _inference clusters_ to serve models at scale with low latency
- _Experiment tracking_ and _model monitoring_ to maintain quality in production

## Conclusion

Slack‘s data architecture has evolved significantly to keep pace with the company‘s explosive growth. By iteratively addressing bottlenecks and rethinking existing approaches, Slack has built a platform that can:

- Ingest and serve _petabytes_ of data with _predictable low latency_
- Process _trillions_ of messages and events per day across _millions_ of users
- Enable delightful user experiences powered by _machine learning_

While there‘s no one-size-fits-all solution, Slack‘s journey offers valuable lessons for any organization dealing with rapid data growth. Some key takeaways:

- _Revisit assumptions_ as scale changes – what works for 100 users likely won‘t for 100,000
- Leverage _incremental gains_ – big improvements often come from aggregating many small optimizations
- _Design for change_ – build architectures and abstractions that can evolve as use cases and requirements shift
- _Invest in observability_ – comprehensive logging, metrics and tracing are essential for operating complex systems
- _Apply AI/ML holistically_ – look for opportunities to enhance every layer of the stack, from infrastructure to end-user experiences

Slack‘s data architecture will continue to evolve and scale to power the next generation of workplace collaboration. With a solid foundation and a culture of continuous improvement, the company is well-positioned to innovate and meet the needs of the most demanding enterprises. Exciting times ahead!

## References

[1] Slack. "Slack Announces Fourth Quarter and Fiscal Year 2021 Results"
 [2] VentureBeat. "How Slack uses Flannel to manage multi-tenant architecture"
 [3] Slack Engineering. "Flannel: An Application-Level Edge Cache to Make Slack Scale"
 [4] Slack Engineering. "Reducing Slack‘s memory footprint"
 [5] Slack Engineering. "Scaling Slack‘s Job Queue – Robustly Handling Billions of Tasks in Milliseconds Using Kafka and Redis"
 [6] InfoQ. "Slack‘s Migration to a Service Mesh Architecture"
 [7] The New Stack. "How Slack Uses a Modified Change Data Capture Pattern to Synchronize Billions of Messages"
 [8] GitHub Engineering. "Why We Chose Kafka At Slack"
 [9] Slack Engineering. "Building the Slack AI Platform"

---

Source: [Scaling Slack‘s Data Architecture to Power Collaboration and AI/ML Innovation](https://33rdsquare.com/slack-data-engineering-design-and-architecture/)