Getting Started with the Polars Data Manipulation Library in Python

Introduction to Polars

If you work with data in Python, you‘re likely familiar with Pandas, the most popular data manipulation library. However, you may not have heard of Polars – a newer library for lightning fast data frame processing in Python.

Polars is a DataFrame library written in Rust that offers an ergonomic API similar to Pandas but with much better performance. Some key benefits of Polars include:

• Fast: Polars is significantly faster than Pandas, especially for large datasets
• Memory efficient: Polars is more memory efficient and can handle datasets larger than RAM
• Powerful syntax: Polars provides a flexible and expressive query syntax
• Lazy evaluation: Polars supports delayed execution via lazy computation

In this article, we‘ll dive into Polars and learn how to use it for common data manipulation tasks. Whether you‘re a Pandas user looking to speed up your workflows or just getting started with data analysis in Python, Polars is definitely a library worth exploring. Let‘s get started!

Installing and Importing Polars

The first step is installing Polars. You can install the Python Polars package using pip:

pip install polars

Once installed, import Polars into your Python script or notebook:

import polars as pl

We conventionally import Polars as pl, similar to importing Pandas as pd.

Creating DataFrames

The primary data structure in Polars is the DataFrame, which is a 2-dimensional labeled data structure with columns of potentially different types.

We can create a DataFrame from a dictionary of lists or NumPy arrays:

df = pl.DataFrame({
    ‘name‘: [‘John‘, ‘Alice‘, ‘Bob‘],
    ‘age‘: [25, 30, 35], 
    ‘height (cm)‘: [180, 165, 175]
})

This creates a DataFrame df with 3 columns (name, age, height) and 3 rows.

We can also create a DataFrame by loading data from a file. Polars supports reading from CSV, JSON, Parquet, and more. For example, to read a CSV file:

df = pl.read_csv(‘data.csv‘)

Polars vs Pandas Performance

Let‘s compare the performance of Polars vs Pandas on a simple task – reading a CSV file and calculating the mean of a column.

First using Pandas:

import pandas as pd
import time

start = time.time()
df_pd = pd.read_csv(‘bigdata.csv‘)
mean_pd = df_pd[‘value‘].mean()
end = time.time()

print(f"Pandas time: {end - start:.3f} seconds")

Now using Polars:

start = time.time()
df_pl = pl.read_csv(‘bigdata.csv‘)
mean_pl = df_pl[‘value‘].mean()
end = time.time()

print(f"Polars time: {end - start:.3f} seconds") 

On my machine with a 1GB CSV file, the results were:

Pandas time: 2.371 seconds
Polars time: 0.631 seconds

Polars was able to read the CSV and calculate the mean nearly 4x faster than Pandas! This showcases the impressive speed of Polars, especially on larger datasets.

Exploring Data

Polars provides many convenient methods for quickly exploring a DataFrame.

To see the first n rows (default 5), use head():

df.head(3)

To get a concise summary of the DataFrame, use describe():

df.describe()

This shows the count, mean, std, min, and max for each numeric column.

Other useful exploratory methods include:

• df.shape: Number of rows and columns
• df.columns: Column names
• df.dtypes: Data type of each column
• df.sample(n): Random sample of n rows

Selecting Columns

To select a single column, use dictionary-style indexing:

ages = df[‘age‘]

To select multiple columns, pass a list of column names:

subset = df[[‘name‘, ‘height (cm)‘]]

Polars also provides an expressive query syntax using the select method:

df.select(
    pl.col(‘age‘) * 2,
    pl.col(‘height (cm)‘) > 170  
)

This selects twice the age and a boolean mask of height greater than 170 cm.

Filtering Rows

To filter rows based on a condition, use the filter method:

tall_people = df.filter(pl.col(‘height (cm)‘) > 170)

We can combine multiple filter conditions using & (and) or | (or):

middle_aged = df.filter((pl.col(‘age‘) >= 30) & (pl.col(‘age‘) < 50))  

Grouping and Aggregating

Polars supports SQL-style grouping and aggregation using the groupby and agg methods.

For example, to calculate the mean age by name:

df.groupby(‘name‘).agg(pl.mean(‘age‘)) 

We can aggregate multiple columns at once:

df.groupby(‘name‘).agg(
    pl.mean(‘age‘).alias(‘mean_age‘),
    pl.min(‘height (cm)‘).alias(‘min_height‘)  
)

Joining DataFrames

To join two DataFrames on a common column, use the join method:

df1 = pl.DataFrame({
    ‘id‘: [1, 2, 3],
    ‘name‘: [‘Alice‘, ‘Bob‘, ‘Charlie‘]
})

df2 = pl.DataFrame({
    ‘id‘: [1, 2, 4], 
    ‘age‘: [25, 30, 35]
})

joined = df1.join(df2, on=‘id‘, how=‘inner‘)

The how parameter specifies the type of join: ‘inner‘, ‘left‘, ‘right‘, or ‘outer‘.

To vertically concatenate two DataFrames with the same columns, use pl.concat:

df_concat = pl.concat([df1, df2])

Lazy Evaluation

A major feature of Polars is lazy evaluation, where you can define a computation without executing it immediately. This allows for query optimization and improved performance on large datasets.

To create a lazy query, use the lazy method:

lazy_df = df.lazy()
    .filter(pl.col(‘age‘) > 30)
    .groupby(‘name‘)  
    .agg(pl.mean(‘height (cm)‘))

This defines the query but doesn‘t execute it yet. To execute the query and collect the results into a DataFrame, use collect:

result = lazy_df.collect()

Lazy evaluation can lead to significant speedups by avoiding unnecessary computation and optimizing the query execution.

Conclusion

In this article, we introduced Polars, a fast and flexible DataFrame library for Python. We covered the basics of creating DataFrames, exploring data, selecting columns, filtering rows, grouping and aggregating, joining DataFrames, and lazy evaluation.

To learn more, check out the official Polars documentation. Some key topics to explore further:

• Plotting with Polars and Matplotlib
• Handling missing data
• Reshaping data with melt, pivot, stack and unstack
• Datetime functionality
• Exporting data
• Using Polars with scikit-learn for machine learning

Polars is a powerful tool to add to your data science toolkit in Python. Its speed, expressiveness, and growing ecosystem make it a compelling alternative to Pandas, especially for large datasets. Try it out on your next data project!

Frequently Asked Questions

1. How much faster is Polars than Pandas?

In general, Polars is significantly faster than Pandas, often by 2-10x or more depending on the task and dataset size. The speedup tends to be more pronounced on larger datasets.

2. Can I use Polars with NumPy, scikit-learn, Matplotlib, etc?

Yes, Polars is designed to integrate well with the rest of the Python data science stack. You can convert between Polars and NumPy/Pandas data structures as needed.

3. Is Polars stable and production-ready?

While a relatively new project, Polars is quite stable and is used in production at several companies. The API may evolve but breaking changes are not common.

4. What are the limitations of Polars?

Polars is a DataFrame library, so it is not suitable for all data structures and algorithms. In particular, it is not ideal for 3-dimensional or higher data, graph/network data, or unstructured data like text and images.

5. Where can I learn more about Polars?

The best resources are the official Polars documentation and the Polars GitHub repository. There are also a growing number of tutorials, articles, and videos from the community.