Eager mode
When it comes to data manipulation and analysis in Python, the Pandas library has reigned supreme for many years. Pandas provides an intuitive and flexible interface for working with structured data, with a wide range of functions for filtering, aggregating, reshaping, and visualizing data. However, a new contender called PyPolars has emerged that aims to challenge Pandas‘ dominance, particularly when it comes to performance on large datasets.
PyPolars is a Python library that provides a DataFrame-like data structure and data manipulation tools similar to Pandas. The key difference is that PyPolars is built on top of the Rust library Polars, which enables extremely fast performance, efficient memory utilization, and built-in support for lazy evaluation and parallel processing.
In this article, we‘ll take an in-depth look at PyPolars and explore how it compares to Pandas in terms of features, performance, and ecosystem support. We‘ll walk through examples of using PyPolars for common data analysis tasks and discuss the scenarios where it may be a compelling alternative to Pandas. Let‘s dive in!
Eager vs Lazy Evaluation
One of the key features that distinguishes PyPolars from Pandas is its support for both eager and lazy evaluation.
With eager evaluation, operations on a DataFrame are executed immediately and return a new DataFrame. This is how Pandas primarily operates. While eager evaluation is intuitive to work with, it can lead to performance issues when chaining together many complex operations, as each intermediate step requires materializing a new DataFrame.
Lazy evaluation, on the other hand, allows you to define a pipeline of operations without actually executing them. The computations are only triggered when you call a method to materialize the result, such as collect(). Lazy mode is powerful because it allows PyPolars to optimize the query execution and reduce total compute time.
Here‘s a simple example contrasting eager and lazy evaluation in PyPolars:
import pypolars as pl
df = pl.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df_eager = df.with_column(
pl.col("A") * 2
)
df_lazy = df.lazy().with_column(
pl.col("A") * 2
)
print(df_lazy) # Prints the query plan
df_result = df_lazy.collect()
print(df_result) # Prints the actual result
With the eager API, the with_column operation is executed immediately. With the lazy API, we first define the computation and then trigger execution with collect().
The ability to seamlessly switch between eager and lazy evaluation modes in PyPolars is a powerful feature for optimizing performance. You can develop and test your code in eager mode for simplicity, and then switch to lazy mode for better performance on larger datasets.
PyPolars API Basics
PyPolars aims to provide an API that is similar to Pandas, making it easy for Pandas users to get started. However, there are some differences in syntax and functionality to be aware of.
Creating a DataFrame in PyPolars is straightforward:
import pypolars as pl
df = pl.DataFrame({
"A": [1, 2, 3],
"B": [4.0, 5.0, 6.0],
"C": [True, False, True],
"D": ["a", "b", "c"]
})
Selecting columns and filtering rows is similar to Pandas:
df_ab = df[["A", "B"]]
df_filtered = df[df["A"] > 1]
Applying functions and aggregations to columns uses the select method and expressive syntax:
df_squared = df.with_column(pl.col("A").map(lambda x: x ** 2).alias("A_squared"))
df_grouped = df.groupby("C").agg(
pl.sum("A").alias("A_sum"),
pl.mean("B").alias("B_mean")
)
Notice that we used with_column and groupby().agg() instead of the Pandas equivalents assign and groupby().agg(). These are some of the syntactical differences to be aware of when using PyPolars.
Overall, the PyPolars API aims to be expressive and concise, with method chaining and column-wise operations as first-class citizens. The syntax may feel a bit more functional and declarative compared to Pandas‘ more object-oriented style.
Performance Benchmarks
The main value proposition of PyPolars is its excellent performance, especially on larger datasets. Let‘s take a look at some benchmarks comparing PyPolars to Pandas.
The PyPolars documentation includes a performance benchmark comparing various operations in PyPolars vs Pandas. Here are a few highlights:
- On a DataFrame with 10 million rows, filtering in PyPolars was 30x faster than Pandas
- Grouping by a single column and summing was 8x faster in PyPolars compared to Pandas
- Joining two DataFrames was 5x faster in PyPolars
In general, the benchmarks show PyPolars outperforming Pandas by a significant margin on larger datasets, sometimes by an order of magnitude. This is due to PyPolars‘ use of the highly optimized Rust library Polars under the hood, as well as its support for lazy evaluation and parallel processing.
It‘s worth noting that for smaller datasets, the performance difference between PyPolars and Pandas may not be as pronounced. Pandas is a mature and optimized library, and it can be very fast for many common operations on small to medium sized data.
However, as data sizes grow and operations become more complex, the benefits of PyPolars‘ design start to shine. If you regularly work with large datasets in the millions or billions of rows, PyPolars is definitely worth considering for its performance advantages.
Ecosystem and Limitations
While PyPolars shows a lot of promise, it‘s important to acknowledge that it is a younger library compared to Pandas. Pandas has a massive ecosystem of tools, extensions, and integrations built around it.
For example, many popular libraries like scikit-learn, statsmodels, and plotnine work seamlessly with Pandas DataFrames. PyPolars is still building out its ecosystem integrations, and not all libraries support PyPolars DataFrames as input yet.
Additionally, there are some features and data types that PyPolars currently lacks compared to Pandas. As of the latest PyPolars release (0.17.3), some notable limitations include:
- No support for text data (Pandas has the StringDtype)
- Limited support for time series and temporal data
- No support for categorical data type
- No native CSV reader (requires loading CSV data with Pandas and converting to PyPolars DataFrame)
The PyPolars team is actively working on addressing these limitations and expanding functionality. Text and datetime support are already on the roadmap for future releases.
Despite these current limitations, PyPolars still covers a wide range of common data manipulation and analysis tasks. For many analytics and data science workflows, PyPolars can be a highly performant alternative to Pandas.
Conclusion
So is PyPolars a viable alternative to Pandas? Based on its impressive performance, intuitive API, and support for lazy evaluation, I believe the answer is yes – for certain use cases.
If you regularly work with large datasets and find yourself running into performance bottlenecks with Pandas, PyPolars is definitely worth evaluating. Its ability to optimize queries and crunch through millions of rows with ease is a major selling point.
On the other hand, if your data fits comfortably in memory and you rely heavily on the broader Pandas ecosystem, the benefits of PyPolars may be less compelling. Pandas remains a powerful and mature library with excellent documentation and community support.
Ultimately, the choice between PyPolars and Pandas depends on your specific needs and priorities. For data-intensive applications that demand the utmost performance and scalability, PyPolars is a very promising option to add to your toolkit.
The PyPolars library has come a long way since its initial release, and it continues to evolve rapidly. With a focused and enthusiastic developer community behind it, I‘m excited to see how PyPolars grows and matures over the coming years.
I encourage you to give PyPolars a try and see how it performs on your own datasets and workflows. The future of data analysis in Python looks bright, and PyPolars is poised to play a major role in it.