Visualizing Data Flow with Sankey Diagrams: A Comprehensive Guide for Python Developers and Data Scientists

As data volumes continue to grow and analytic complexity increases, data scientists are constantly seeking more effective ways to make sense of interconnected, dynamic systems. One powerful tool for visualizing data flow relationships is the Sankey diagram.

Sankey diagrams have a long history, dating back to the late 19th century when Irish engineer Matthew Sankey used them to illustrate energy flows in steam engines. His original diagrams used arrows of varying widths to show the quantity of energy flow between different parts of the engine, demonstrating visually how much of the input energy was output as useful work versus lost as heat.

Since then, Sankey diagrams have been used to visualize flows of all kinds, from material and energy in physical systems, to traffic between locations, to money through financial accounts. In recent years, Sankey diagrams have found new life in the world of data visualization, becoming an increasingly popular choice for making complex data flows intelligible.

Why Use Sankey Diagrams?

The power of Sankey diagrams lies in their ability to simultaneously represent multiple key properties of flow data in an intuitive way:

The direction of flow from source to target
The magnitude of flow quantities
The proportion of flows relative to each other
The categorization of nodes into meaningful groups

By encoding this information graphically with node positioning and link widths, Sankey diagrams translate data tables into a picture worth a thousand words. They enable viewers to quickly grasp the dominant flow paths, spot bottlenecks and inefficiencies, and understand overall system dynamics.

Some specific advantages of Sankey diagrams for data visualization include:

Efficient use of space to display flows between many nodes without excessive clutter or occlusion
Ability to show hierarchical or multi-stage flows across several sequential node layers or steps
Flexibility to visualize both conserved flows (where total input always equals total output) and non-conserved flows (where quantities can be lost or gained)
Intuitive readability even for non-technical audiences – the thicker the link, the greater the flow

Limitations of Sankey Diagrams

Of course, no visualization is perfect for every situation. Some notable limitations and challenges with Sankey diagrams include:

Link crossings can become visually confusing in complex multi-layer diagrams with many nodes
Flows that diverge and re-converge between the same nodes at different stages can be hard to trace
Sankey diagrams are poorly suited for cyclical or recurring flows that double back on themselves
Flows between more than a few categorical node layers may require aggregation to remain intelligible
Interactivity is often needed for viewers to explore large Sankey diagrams with dozens or hundreds of nodes and links

Thus, while Sankey diagrams are a go-to choice for many kinds of data flow visualization, it‘s important to recognize when a different technique like a chord diagram or network graph might be more appropriate.

Sankey Diagram Examples

To better understand the applications and impact of Sankey diagrams, let‘s look at a few real-world examples from different industries and domains:

Energy Flow Visualization

One of the most common modern applications of Sankey diagrams is in visualizing energy production and consumption flows. Agencies like the U.S. Energy Information Administration (EIA) frequently use interactive Sankey diagrams to show how primary energy sources flow through intermediate processing to end uses.

Source: U.S. Energy Information Administration

EIA‘s 2019 U.S. energy flow Sankey diagram provides a comprehensive snapshot of the nation‘s energy system. Key insights pop out visually, such as the dominance of petroleum in the transportation sector, or the significant share of energy rejected as waste heat in electricity generation. Imagine trying to glean the same insights from a table of energy production and consumption statistics!

Google Analytics Customer Flow

Another domain where Sankey diagrams shine is in web analytics, where they‘re used to visualize visitor flow through a website or app. For example, Google Analytics offers Sankey-style "customer flow" reports that show how users navigate between pages, and where they drop off.

Source: ConversionXL

This type of flow visualization helps digital marketers and UX designers pinpoint high-traffic pages, spot navigation bottlenecks, and optimize conversion funnels. What previously might have required piecing together data from separate pageview and event reports can now be grasped at a glance.

Manufacturing Process Flow

In industrial engineering and operations management, Sankey diagrams are often used to analyze manufacturing process flows and optimize production lines. Flows of raw materials, work in progress, and finished goods between workstations or production stages can be quantified and visualized to uncover waste and inefficiency.

Source: Wikimedia Commons

This simplified example shows how a Sankey diagram can highlight key metrics like scrap rate, inventory, and throughput at different steps in a manufacturing process. More complex production environments with multiple products, machines, and routings can also be visualized through multi-layer Sankey diagrams.

How to Create Sankey Diagrams in Python

Now that we‘ve seen the power of Sankey diagrams for visualizing flow data, let‘s dive into how data scientists and developers can create them programmatically in Python. While it‘s possible to manually create Sankey diagrams in tools like Adobe Illustrator or D3.js, generating them automatically from data using code has major advantages:

Flexibility to rapidly explore different node definitions, groupings, and level of detail
Ability to create dynamic, interactive diagrams that change based on user input or real-time data
Reproducibility and integration with the rest of your Python data pipeline and infrastructure

The main Python libraries and packages that support creating Sankey diagrams include:

Plotly (as seen in the examples above) – Provides a high-level Sankey class for defining nodes and links
Matplotlib – Lower-level plotting library that can draw Sankey diagrams with the Sankey function in the matplotlib.sankey module
Holoviews – Declarative visualization library that supports Sankey diagrams as part of its "Network" elements
iPySankey – Jupyter widget for interactive Sankey diagrams rendered with D3 in the notebook

For the rest of this section, we‘ll focus on creating Sankey diagrams with Plotly, since it offers an intuitive, well-documented API suitable for both beginners and advanced users. But the same principles apply to the other libraries as well.

Basic Sankey Diagram with Plotly

To create a basic Sankey diagram with Plotly, you need to define two main elements:

Nodes – The entities that act as the sources and targets for flow in the diagram. Defined by a label (required) and optional properties like color and x/y position.
Links – The flows between nodes in the diagram. Defined by a source and target node index, a flow value, and optional properties like color and label.

Here‘s a minimal example of creating a Sankey diagram of supermarket sales data in Plotly:

import plotly.graph_objects as go

nodes = {
  "label": ["Farm", "Factory", "Warehouse", "Supermarket"], 
  "color": ["#8dd3c7","#ffffb3","#bebada","#fb8072"] 
}  

links = {
  "source": [0,0,1,1,2], 
  "target": [1,2,2,3,3], 
  "value":  [20,10,30,15,35]  
}

sankey = go.Sankey(node=nodes, link=links) 
fig = go.Figure(sankey) 
fig.show()

This code defines a simple food supply chain with four node types (Farm, Factory, Warehouse, Supermarket) and five links representing flows of goods between them. The resulting diagram isn‘t exactly production-ready, but it shows how easily you can get a Sankey visualization from just a few lines of Python.

Advanced Sankey Diagram Techniques with Plotly

Of course, most real-world Sankey diagrams are quite a bit more complex than the basic example above, with many more nodes and links, customized appearance, and interactive behavior. Plotly provides a number of configuration options to create production-quality Sankey diagrams for nearly any use case.

Let‘s look at a more advanced example that visualizes data on airline flight delays by cause and carrier, using a dataset from the U.S. Bureau of Transportation Statistics:

import plotly.graph_objects as go
import pandas as pd

## Load and preprocess data
df = pd.read_csv(‘airline_delay_causes.csv‘)
carriers = df.groupby(‘carrier‘)[‘arr_flights‘].sum().reset_index()
carriers = carriers.sort_values(‘arr_flights‘, ascending=False).head(10)
df = df[df.carrier.isin(carriers.carrier)]
df[‘total_delay‘] = df[‘weather_delay‘] + df[‘nas_delay‘] + 
                    df[‘security_delay‘] + df[‘late_aircraft_delay‘]

## Create Sankey Diagram 
nodes = {
  ‘label‘: list(carriers.carrier) + 
           [‘Weather‘,‘NAS‘,‘Security‘,‘Late Aircraft‘,‘Canceled‘], 
  ‘color‘: ["#8dd3c7","#ffffb3","#bebada","#fb8072","#80b1d3","#fdb462",
            "#b3de69","#fccde5","#d9d9d9","#bc80bd"] +
           ["#ccebc5","#ffed6f","#fb9a99","#a6cee3","#984ea3"]
}

weather =     list(df[‘weather_delay‘])
nas =         list(df[‘nas_delay‘])
security =    list(df[‘security_delay‘])
late_aircraft = list(df[‘late_aircraft_delay‘])
canceled =    list(df[‘cancelled‘])

sources = list(range(len(carriers))) * 5
targets = [10] * len(carriers) + [11] * len(carriers) + [12] * len(carriers) + \
          [13] * len(carriers) + [14] * len(carriers)          
values =  weather + nas + security + late_aircraft + canceled

links = {
  ‘source‘: sources, 
  ‘target‘: targets, 
  ‘value‘:  values
}

sankey = go.Sankey(
  node=nodes,
  link=links,
  arrangement=‘snap‘,
  valueformat=‘.0f‘, 
  valuesuffix=‘ flights‘
)  

fig = go.Figure(sankey)

fig.update_layout(
  title_text=‘Airline Flight Delays by Cause and Carrier‘, 
  font_size=12
)

fig.show()

This code demonstrates several advanced features and best practices for creating effective Sankey diagrams with Plotly:

Data preprocessing – Filtering and aggregating raw data into a suitable format for visualization, including calculated fields like total_delay
Node definition – Using custom label and color lists to define meaningful, visually distinct node groupings
Link definition – Constructing source, target, and value lists to connect nodes based on delay metrics and encode flow quantities
Value formatting – Setting a custom number valueformat and valuesuffix for readable link hover labels
Layout customization – Adjusting diagram arrangement to minimize link crossings, setting title font_size for readability

The end result is a richly detailed Sankey diagram that allows viewers to quickly identify the top delay causes by airline, compare delay proportions across carriers, and grasp the relative magnitude of total delays. Achieving this level of insight with a static dataset would be much more challenging.

With the built-in interactivity of Plotly Sankey diagrams, users can hover over nodes and links to see exact values, pan and zoom the diagram, and even select nodes to highlight and isolate sub-flows. All of this functionality is available automatically, without having to write a single line of JavaScript or HTML.

Conclusion: Advancing the Art and Science of Data Flow Visualization

As data pipelines grow in complexity, and businesses seek ever-deeper insights into dynamic flow processes, Sankey diagrams will undoubtedly continue to grow in importance as a visualization tool. For data scientists and Python developers, the key to creating effective Sankey diagrams is to:

Understand the strengths and limitations – Know when a Sankey diagram is the right choice given the data and questions at hand, and when an alternative approach would work better.
Choose the right tools – Whether it‘s Plotly, Matplotlib, or another Python library, pick a Sankey diagram implementation that balances ease of use with customization options.
Master the data manipulation workflow – Structure your raw data and build a reproducible pipeline for generating meaningful nodes and links across multiple categorical dimensions.
Refine the visual design – Experiment with features like node colors and positions, diagram layouts, and interactivity to highlight key insights and guide the viewer‘s attention.
Iterate and share – Treat your Sankey diagrams as an evolving part of your data storytelling practice, and don‘t hesitate to collect feedback and try new approaches.

Whether you‘re a beginner just learning to create your first Sankey diagrams, or an experienced practitioner pushing the boundaries of interactive flow visualization, I hope this guide has equipped you with the context and practical advice you need to uplevel your skills.

As we‘ve seen, Sankey diagrams are a powerful way to transform complex flow data into an intuitive, meaningful picture. They bridge the gap between the intricate workings of real-world systems and the human need for accessible, actionable insights.

By following the principles and techniques laid out here, you‘ll be well on your way to creating Sankey diagrams that inform, engage, and inspire your audience. So go forth and visualize! The world of data flow awaits.

Visualizing Data Flow with Sankey Diagrams: A Comprehensive Guide for Python Developers and Data Scientists

Why Use Sankey Diagrams?

Limitations of Sankey Diagrams