10 Lesser-Known Pandas Functions You Should Know in 2026

Introduction

When it comes to data analysis and manipulation in Python, pandas is the go-to library for most data scientists and analysts. While functions like read_csv, groupby, and merge are well-known and frequently used, pandas offers a wealth of other powerful functions that can make your data wrangling tasks easier and more efficient.

In this article, I want to introduce you to 10 lesser-known but incredibly useful pandas functions that you should consider adding to your data science toolkit in 2023. These functions can help with everything from handling missing values and filtering data to reshaping your DataFrame and accessing specific values.

Whether you‘re a pandas novice or a seasoned pro, learning these functions will undoubtedly level up your data wrangling skills. So let‘s dive in and explore these hidden gems of the pandas library!

1. ffill and bfill

When working with time series data, it‘s common to encounter missing values. While there are many strategies for imputing missing values, forward filling (ffill) and backward filling (bfill) are two simple yet effective methods that are particularly useful when your data has a meaningful order, such as timestamps.

The ffill method propagates the last known non-null value forward until the next non-null value is encountered. Conversely, bfill works in the opposite direction, propagating the next known non-null value backward. Here‘s a quick example:

import pandas as pd
df = pd.DataFrame({‘date‘: pd.date_range(start=‘1/1/2023‘, periods=5),
‘value‘: [1, np.nan, np.nan, 4, 5]})
df.ffill()

This would produce:

        date  value
0 2023-01-01    1.0
1 2023-01-02    1.0
2 2023-01-03    1.0
3 2023-01-04    4.0
4 2023-01-05    5.0

As you can see, the missing values were filled with the last known value. This is useful when you want to carry forward the most recent observation, such as in stock price data where the last known price is often the best estimate for any missing prices.

Some common use cases for ffill and bfill include:

• Filling missing sensor readings in IoT data streams
• Imputing missing stock prices in financial data
• Handling missing measurements in scientific experiments

2. shift

The shift function is another handy tool for working with time series data. It allows you to shift the index of your DataFrame or Series by a specified number of periods, which is useful for calculating lagged values or differences over time.

Here‘s an example of using shift to calculate the one-period change in a stock price:

df[‘price_change‘] = df[‘price‘] - df[‘price‘].shift(1)

This creates a new column price_change that contains the difference between the current price and the previous day‘s price.

Some other common applications of shift include:

• Creating lagged features for time series forecasting models
• Calculating rolling averages or moving windows
• Comparing values across different time periods

3. select_dtypes

When working with a DataFrame that contains columns of mixed data types, it can be helpful to filter the columns based on their dtype. That‘s where the select_dtypes function comes in handy.

For example, let‘s say you have a DataFrame with columns of type int64, float64, and object (string), but you only want to work with the numeric columns. You could use select_dtypes like this:

numeric_cols = df.select_dtypes(include=[‘int64‘, ‘float64‘])

This would return a new DataFrame containing only the columns of type int64 and float64.

Alternatively, you can use select_dtypes to exclude certain data types:

  
non_numeric_cols = df.select_dtypes(exclude=[‘int64‘, ‘float64‘])

Some situations where select_dtypes is particularly useful:

• Separating categorical columns from numeric columns for different preprocessing steps
• Ensuring your model inputs are all of the expected data types
• Exploring the distribution of different data types in your DataFrame

4. clip

The clip function allows you to limit the values in your DataFrame or Series to a specified range. This is useful for handling outliers or capping values at a certain threshold.

For example, let‘s say you have a DataFrame with a column called ‘age‘ and you want to limit the ages to between 0 and 120. You could use clip like this:

df[‘age‘] = df[‘age‘].clip(lower=0, upper=120)  

Any values less than 0 would be set to 0, and any values greater than 120 would be set to 120.

Some other potential use cases for clip:

• Capping sensor readings at the maximum possible value
• Ensuring input features are within the expected range for a model
• Limiting prices to a certain range for an e-commerce application

5. query

The query function provides a concise way to filter a DataFrame based on a boolean expression. It‘s similar to using boolean indexing, but often more readable and convenient, especially for complex conditions.

For example, instead of writing:

filtered_df = df[(df[‘age‘] > 18) & (df[‘country‘] == ‘USA‘)]  

You could use query like this:

filtered_df = df.query(‘age > 18 and country == "USA"‘)

This is much more intuitive and easier to read, especially as your filtering conditions get more complex.

Some situations where query shines:

• Filtering a large DataFrame based on multiple conditions
• Dynamic filtering based on user input or variables
• Improving code readability for complex boolean indexing

6. melt

The melt function is used to reshape a DataFrame from wide format to long format. This is a common data preprocessing step, especially when working with categorical data or time series data.

For example, let‘s say you have a DataFrame with columns for different years:

df = pd.DataFrame({‘name‘: [‘Alice‘, ‘Bob‘, ‘Charlie‘],
                   ‘2020‘: [1, 2, 3], 
                   ‘2021‘: [4, 5, 6],
                   ‘2022‘: [7, 8, 9]})

To reshape this into long format, you could use melt like this:

melted_df = df.melt(id_vars=[‘name‘], 
                    var_name=‘year‘, 
                    value_name=‘value‘)

This would produce:

     name  year  value
0   Alice  2020      1
1     Bob  2020      2
2  Charlie 2020      3
3   Alice  2021      4
4     Bob  2021      5
5  Charlie 2021      6
6   Alice  2022      7
7     Bob  2022      8
8  Charlie 2022      9

As you can see, the DataFrame is now in long format with columns for name, year, and value. This format is often easier to work with for certain types of analysis and visualization.

Some common use cases for melt:

• Reshaping wide time series data into long format for plotting or analysis
• Converting categorical variables from wide format to long format for encoding
• Preparing data for input into certain types of statistical models

7. where

The where function is used to conditionally replace values in a DataFrame or Series based on a boolean mask. It‘s a more concise and efficient alternative to using loc or boolean indexing for conditional replacement.

For example, let‘s say you want to replace all negative values in a DataFrame with 0. You could use where like this:

df = df.where(df >= 0, 0)

This replaces any values less than 0 with 0, while leaving the rest of the values unchanged.

You can also use where with a callable function for more complex conditions:

df = df.where(lambda x: x % 2 == 0, -1)

This replaces any odd values with -1, while leaving even values unchanged.

Some situations where where is handy:

• Replacing invalid or missing values based on a condition
• Encoding categorical variables based on a complex condition
• Filtering a DataFrame in-place based on a condition

8. iat

The iat function provides a fast and concise way to access a single value in a DataFrame by its integer position. It‘s similar to using iloc, but only for accessing a single value rather than a slice.

For example, to access the value in the 3rd row and 2nd column of a DataFrame, you could use iat like this:

value = df.iat[2, 1]  

This is more concise and slightly faster than using iloc:

value = df.iloc[2, 1]

Some situations where iat is useful:

• Accessing a specific value in a large DataFrame
• Updating a single value in a DataFrame
• Iterating over DataFrame values in a fast and memory-efficient way

Conclusion

In this article, we‘ve explored 8 lesser-known but powerful pandas functions that can make your data wrangling tasks easier and more efficient. From handling missing values with ffill and bfill, to reshaping data with melt, to conditional replacement with where, these functions cover a wide range of common data preprocessing tasks.

By adding these functions to your pandas toolkit, you‘ll be able to write cleaner, more concise, and more efficient code. You‘ll also be able to tackle a wider range of data wrangling challenges with ease.

Of course, this is just a small sample of the many useful functions that pandas has to offer. I encourage you to explore the pandas documentation and experiment with these and other functions on your own datasets. With practice and experience, you‘ll develop a strong intuition for when and how to use these tools to streamline your data analysis workflow.

Remember, the key to mastering pandas is not just knowing the functions, but understanding how to apply them in the context of real-world data challenges. So get out there and start wrangling some data! And if you have any favorite lesser-known pandas functions of your own, feel free to share them in the comments below.

Happy data wrangling!