Taming Large Datasets: How to Dramatically Reduce Memory Usage in Python Pandas
As data scientists and analysts, we often find ourselves working with very large datasets that push the limits of our system‘s memory. When you try to load a multi-gigabyte CSV file into a pandas DataFrame, you may see your memory usage shoot up and start causing issues like sluggish performance or out-of-memory crashes. The good news is that pandas provides several powerful techniques that can dramatically reduce the memory footprint of your DataFrames. In this post, we‘ll dive into exactly how you can implement these in your own projects.
Why is DataFrame memory usage important?
Before we get to the solutions, it‘s worth taking a moment to understand the problem. A pandas DataFrame is essentially a 2-dimensional labeled data structure with columns of potentially different types. Under the hood, pandas stores the data for each DataFrame column as a separate NumPy array.
While the DataFrame abstraction makes it very convenient to work with tabular data in Python, as your data gets larger, the memory usage can start to become a major concern. If you‘re not careful, you can easily run out of memory, even on a powerful machine. This is because pandas defaults to using high-precision numeric types like 64-bit floats and integers, and the flexible object type for strings. These can use up much more memory than is actually required for your particular dataset.
Luckily, pandas provides great tools to help you understand your memory usage. Let‘s load in an example DataFrame to see how this works:
import pandas as pd
import numpy as np
df = pd.DataFrame({‘col1‘: np.random.rand(1000000),
‘col2‘: np.random.randint(0, 10, 1000000),
‘col3‘: [‘category‘+str(i) for i in np.random.randint(0, 5, 1000000)]})This creates a DataFrame with a million rows and 3 columns of different dtypes: float, int, and object.
Checking DataFrame Memory Usage
The .info() method provides a quick way to see the dtypes and non-null counts of each column, as well as the total memory usage:
df.info()
<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 1000000 entries, 0 to 999999
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 col1 1000000 non-null float64
1 col2 1000000 non-null int64
2 col3 1000000 non-null object
dtypes: float64(1), int64(1), object(1)
memory usage: 22.9+ MBHere we can see that the DataFrame is using about 23 MB of memory, with the bulk coming from the float64 and object columns.
For even more detail on a per-column basis, use the .memory_usage() method:
df.memory_usage(deep=True)
Index 128
col1 8000000
col2 8000000
col3 32000000
dtype: int64The deep=True option gives an accurate introspection of the memory usage. We can see the object column using the most at 32 MB, followed by 8 MB each for the float and integer columns. Those 64-bit dtypes add up quickly with a million rows!
Reducing DataFrame Memory Usage
Now that we understand the problem, let‘s look at some solutions. The key idea is to use more memory-efficient dtypes wherever possible.
1. Downcasting Numerical Dtypes
In many cases, the full precision of a 64-bit float or integer is overkill for the actual data at hand. Pandas allows you to easily downcast to smaller dtypes:
df[‘col1‘] = df[‘col1‘].astype(‘float32‘)
df[‘col2‘] = df[‘col2‘].astype(‘int8‘)
df.info()
<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 1000000 entries, 0 to 999999
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 col1 1000000 non-null float32
1 col2 1000000 non-null int8
2 col3 1000000 non-null object
dtypes: float32(1), int8(1), object(1)
memory usage: 13.4+ MBBy downcasting to a 32-bit float and 8-bit integer, we‘ve reduced the DataFrame memory usage from 23 MB to 13 MB, a savings of over 40%! The float32 dtype still provides plenty of precision in most cases. And since the integers in col2 only ranged from 0 to 10, an 8-bit signed integer was sufficient to represent them.
When downcasting, always make sure the smaller dtype has an appropriate range for your data to avoid losing information. Pandas will raise an error if downcastingg causes issues.
2. Converting Object Columns to Categories
String or object columns often take up a large amount of memory, especially when there are many repeated values. The pandas Categorical dtype can store the data much more efficiently in these cases.
Let‘s convert our object column to a category:
df[‘col3‘] = df[‘col3‘].astype(‘category‘)
df.info()
<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 1000000 entries, 0 to 999999
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 col1 1000000 non-null float32
1 col2 1000000 non-null int8
2 col3 1000000 non-null category
dtypes: category(1), float32(1), int8(1)
memory usage: 3.8 MBWow, we‘ve now reduced the total memory usage from 23 MB to under 4 MB, a savings of over 83%! The Categorical dtype stores the unique string values separately and then uses integer codes to represent each value, resulting in huge memory savings.
3. Using Nullable Integer Dtypes
We can get even more memory savings by using the nullable integer dtypes introduced in pandas 1.0. These allow for representing missing values more efficiently compared to using separate float dtypes.
df[‘col2‘] = df[‘col2‘].astype(‘Int8‘)
df.info()
<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 1000000 entries, 0 to 999999
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 col1 1000000 non-null float32
1 col2 1000000 non-null Int8
2 col3 1000000 non-null category
dtypes: Int8(1), category(1), float32(1)
memory usage: 2.3 MBBy using a nullable integer dtype, we‘ve gotten the total memory usage down to just 2.3 MB, about a 90% reduction from where we started!
Of course, the exact memory savings you‘ll see depends on the specifics of your dataset, but these techniques are broadly applicable. I‘d encourage you to profile the memory usage of your key DataFrames and experiment with downcast, Categorical, and nullable integer dtypes.
Additional Memory Optimization Techniques
In addition to using memory-efficient dtypes, there are a few other techniques worth mentioning for optimizing memory usage with pandas:
-
Only load the columns you need: When reading in data from a file, you can specify the
usecolsparameter to load only a subset of columns. This can greatly reduce memory usage if you have datasets with hundreds of columns but only need to work with a few. -
Process the data in chunks: The
chunksizeparameter lets you iterate through chunks of a file one at a time rather than loading the entire dataset into memory at once. Performing operations on each chunk and aggregating the results can allow you to work with datasets that are larger than memory. -
Use Dask for parallel/distributed processing: For even larger datasets that exceed memory even when chunked, the Dask library provides DataFrame and Series abstractions that can work with datasets in parallel across multiple cores or machines. Dask integrates well with the pandas API.
Conclusion
In this post, we‘ve taken a deep dive into the topic of reducing memory usage for pandas DataFrames. We started by looking at how to measure memory usage with the .info() and .memory_usage() methods.
Then we explored three key techniques for optimizing DataFrame memory usage:
- Downcasting numerical dtypes
- Converting object columns to Categorical dtypes
- Using nullable integer dtypes
Through examples, we saw how these techniques can dramatically reduce memory usage, in some cases by 90% or more! We also touched on some additional strategies like loading only necessary columns and processing data in chunks.
As data scientists, being able to efficiently analyze datasets that push the limits of memory is a critical skill. Add these tools to your pandas toolkit and you‘ll be well equipped to tackle even the biggest data challenges.
Here are some resources for further reading:
- Pandas User Guide on Categorical Data: https://pandas.pydata.org/docs/user_guide/categorical.html
- Blog post on Pandas Memory Optimization: https://www.dataquest.io/blog/pandas-big-data/
- Intro to using Dask with Pandas: https://docs.dask.org/en/latest/dataframe.html
Happy data wrangling!
