Why Every Data Scientist Should Avoid Python Lists

Python lists are often the go-to data structure for data scientists due to their simplicity and flexibility. With just a pair of square brackets, you can create a list to store any kind of sequence or collection:

primes = [2, 3, 5, 7, 11]
animals = [‘cat‘, ‘dog‘, ‘rabbit‘]
data = [1, ‘apple‘, 3.14, [‘a‘, ‘list‘, ‘inside‘, ‘a‘, ‘list‘]]

This versatility makes lists an appealing choice, especially for those coming from other programming languages or who are new to Python. However, Python lists come with significant memory and performance overhead that can cause problems as your datasets grow larger.

In this post, we‘ll look under the hood to see why Python lists are so inefficient compared to other data structures. Then we‘ll benchmark some alternatives to show how you can speed up your data processing and analysis by avoiding lists. Finally, we‘ll discuss some strategies for working with mixed data types without resorting to lists.

Python Lists Have a Lot of Overhead

To understand why Python lists are inefficient for large amounts of data, we first need to understand a bit about how Python manages memory. When you create any Python object, the interpreter allocates a chunk of memory to store the object‘s data along with additional metadata like its type, size, and reference count. We can inspect this using Python‘s builtin sys module:

import sys

x = 1.23
print(sys.getsizeof(x)) 
> 24

s = ‘hello world‘
print(sys.getsizeof(s))
> 60

A single Python float consumes 24 bytes, while a string of 11 characters takes 60 bytes! In both cases, only a small portion of the memory is used to store the actual data (8 bytes for a float, 11 bytes for the string), while the rest is overhead used by Python‘s memory management system.

This overhead applies to every element of a Python list individually. We can use the pympler package to measure the total size of a list, including its elements:

from pympler import asizeof

ints = list(range(1000))
print(asizeof.asizeof(ints))
> 9112

floats = [i/10 for i in range(1000)]
print(asizeof.asizeof(floats)) 
> 9112

strings = [str(i) for i in range(1000)]
print(asizeof.asizeof(strings))
> 49704

To store 1000 integers, floats and strings, Python lists consume about 9KB, 9KB and 49KB of memory respectively. If we stored these as packed C arrays, they would only require 4KB, 8KB and approximately 4KB (assuming 4 bytes per string element). So Python lists use anywhere from 2-12x more memory than needed!

Visually, this is because a Python list stores an array of pointers to individual Python objects, each with their own data and metadata:

In contrast, an array in a language like C stores its data in one dense, contiguous block of memory:

This not only saves a significant amount of memory, but also leads to better performance through locality of reference. With Python lists, the data is scattered randomly throughout memory, while C arrays keep data close together, making it faster to access.

More Efficient Data Structures

For sequences of numeric data, Python offers several alternatives that store data more compactly and efficiently than lists. If you‘re coming from another programming language, using Python‘s builtin array module is a good place to start:

from array import array

arr = array(‘d‘, [i/10 for i in range(1000)])  # ‘d‘ == double precision float
print(sys.getsizeof(arr))
> 8096

The array.array type provides efficient storage of homogeneous data types by packing them in contiguous memory without any per-object overhead. We can use the timeit module to compare the runtime of some simple mathematical operations:

from timeit import timeit

ints = list(range(100000))
arr = array(‘i‘, ints)

print(timeit(lambda: sum(ints), number=100))
> 1.3823773860000808

print(timeit(lambda: sum(arr), number=100))  
> 0.37147184300011755

The array version runs nearly 4x faster than the list by avoiding the overhead of Python‘s memory management and leveraging faster low-level instructions.

However, array.array only supports one-dimensional sequences of primitive types. For more advanced use cases, the NumPy library provides a powerful N-dimensional array object with bindings to optimized C and Fortran routines:

import numpy as np

nparr = np.arange(100000)

print(sys.getsizeof(nparr))  
> 800096

print(timeit(lambda: np.sum(nparr), number=100))
> 0.0497010669997991

NumPy arrays consume almost exactly the same amount of memory as a C array and take advantage of SIMD instructions and parallelization to achieve orders of magnitude speedups over Python loops and lists. On my machine, summing the NumPy array is over 20x faster than summing a list!

Here‘s a table summarizing the runtime of some common operations on lists vs NumPy arrays for a dataset of 10 million random floats:

NumPy achieves these speedups through a combination of techniques:

1. Contiguous memory layout that avoids cache misses and allows SIMD parallelization
2. Offloading computations to optimized C and Fortran routines
3. Avoiding the overhead of Python‘s memory management and garbage collection
4. Providing a high-level API for vectorized operations that avoids slow Python loops

For data science and machine learning applications, using NumPy arrays instead of lists is essential for good performance. Many higher-level data science libraries like Pandas, scikit-learn, TensorFlow, and PyTorch all use NumPy arrays under the hood.

Strategies for Avoiding Python Lists

We‘ve seen how using specialized data structures like arrays and NumPy provide significant performance and memory usage benefits over Python lists for numerical data. But what about situations where you need to store more complex, heterogeneous data?

One strategy is to use Python‘s struct module to pack multiple data types into a single byte buffer, similar to a C struct. You can then store these structs in an array for efficient storage and access. Here‘s an example of storing mixed-type records using structs:

from struct import Struct

record_type = Struct(‘i10s?f‘)  # int, 10-char string, bool, float

records = [
    (1, ‘apples‘.encode(‘utf-8‘), True, 1.99),
    (2, ‘bananas‘.encode(‘utf-8‘), False, 0.99),
    (3, ‘oranges‘.encode(‘utf-8‘), True, 3.99)
]

data = bytearray()
for rec in records:
    data.extend(record_type.pack(*rec))

print(sys.getsizeof(data))  
> 120

The packed bytearray uses only 120 bytes to store 3 records, compared to 336 bytes for an equivalent list of tuples. You can also unpack and access individual fields from the bytearray as needed.

If you need a more full-featured solution, the Pandas library provides a DataFrame object that can efficiently store mixed data types in a tabular format. DataFrames use NumPy arrays internally for columns of homogeneous data, while falling back to Python objects for mixed types.

For extremely large datasets that exceed your machine‘s memory, libraries like Vaex and Dask provide distributed versions of DataFrames that can efficiently process data out-of-core or across multiple machines.

Finally, if absolute top performance is needed, Cython is an optimizing static compiler that extends Python with C-like data types and the ability to declare static types. This allows the compiler to generate more efficient code and avoid many of the dynamic aspects of Python that lead to overhead.

Key Takeaways

To summarize, here are the key points to remember about Python lists and their alternatives:

• Python lists are very flexible, but suffer from significant memory overhead and slowdowns due to dynamic memory management
• For large arrays of numeric data, using NumPy or array.array is much more efficient in both memory usage and performance
• NumPy should be your go-to for numerical computing and data analysis in Python due to its optimized memory layout, vectorized operations, and integration with the broader scientific Python ecosystem
• For heterogeneous data, try to use structs or Pandas DataFrames to compactly store data, or explore out-of-core libraries like Vaex and Dask for very large datasets
• Cython provides a way to optimize critical code paths and avoid many of the dynamic aspects of Python that cause overhead, at the cost of increased development time

As a data scientist, being aware of the performance characteristics of Python‘s builtin data structures and libraries is crucial for writing efficient, scalable code. While lists are a convenient and beginner-friendly choice, their performance limitations make them ill-suited for the large amounts of numerical data encountered in real-world ML and AI applications. By understanding how your data is laid out in memory and leveraging tools like NumPy, you can avoid many common performance pitfalls and write cleaner, faster Python!