20 Must-Know Pandas Functions for Exploratory Data Analysis

Exploratory data analysis (EDA) is a crucial first step in any data science project. Before diving into machine learning or statistical modeling, it‘s essential to first understand the data – its shape and structure, distributions of variables, presence of missing values or outliers, and relationships between features.

Pandas, the powerful data manipulation library for Python, provides a comprehensive set of tools for performing EDA quickly and efficiently. With pandas, you can easily load, filter, reshape, aggregate, merge and visualize data to extract valuable insights.

In this post, we‘ll cover 20 key pandas functions that every data scientist should know for exploratory data analysis. For each function, we‘ll explain what it does, why it‘s useful, and provide code examples. Let‘s dive in!

1. head() / tail()

The head() and tail() functions allow you to quickly preview the first or last few rows of a DataFrame. This is useful for getting a sense of what the data looks like and the types of values in each column.

import pandas as pd

df = pd.read_csv(‘data.csv‘)

# Display first 5 rows
df.head()  

# Display last 3 rows  
df.tail(3)

By default, head() displays the first 5 rows, but you can pass an integer to specify the number of rows to show. Similarly, tail() shows the last 5 rows by default.

2. info()

The info() function provides a concise summary of a DataFrame, including the number of rows, column names and data types, and amount of non-null values in each column.

df.info()

Output:

<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 5 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   A       1000 non-null   float64
 1   B       990 non-null    float64
 2   C       1000 non-null   int64  
 3   D       1000 non-null   object 
 4   E       1000 non-null   bool   
dtypes: bool(1), float64(2), int64(1), object(1)
memory usage: 39.2+ KB

This provides a helpful overview of the data and can identify potential issues to investigate further, such as columns with many missing values.

3. describe()

To get basic descriptive statistics on numerical columns like mean, standard deviation, minimum and maximum values, use describe():

df.describe()

Output:

               A          B           C
count  1000.0000  990.00000  1000.00000
mean      0.0214    0.02483   250.50000
std       1.0128    1.00766   144.48141
min      -3.0007   -2.98069     1.00000
25%      -0.6900   -0.69037   125.75000
50%       0.0166    0.02379   250.50000
75%       0.7119    0.72642   375.25000
max       3.2717    2.95158   500.00000

By default this only includes numeric columns. To include all columns, pass include=‘all‘. For categorical variables, it will show frequency counts instead.

4. value_counts()

To see frequency counts of unique values in a column, use value_counts(). This is especially useful for understanding distributions of categorical variables.

df[‘D‘].value_counts()

Output:

red       350
blue      325
green     325
Name: D, dtype: int64

You can normalize the counts to fractions or percentages by passing normalize=True.

5. unique() / nunique()

The unique() and nunique() functions help identify distinct values in a column. unique() returns a list of all unique values, while nunique() returns the number of unique values.

# List all unique values in column D
df[‘D‘].unique()

# Count number of unique values in column D  
df[‘D‘].nunique()

Output:

array([‘red‘, ‘green‘, ‘blue‘], dtype=object)

3

These are helpful for understanding the range of values in a column and spotting potential data quality issues like misspellings or inconsistent capitalization in categorical variables.

6. isnull() / notnull()

To detect missing values, use isnull() and notnull(). These return boolean masks indicating which values are missing or present.

# Count number of missing values in each column
df.isnull().sum()

# Filter rows where column B is not null 
df[df[‘B‘].notnull()]

Inspecting and appropriately handling missing data is an important part of EDA, as many machine learning algorithms don‘t work with missing values.

7. dropna() / fillna()

To remove or fill missing values, use dropna() and fillna() respectively. dropna() will remove rows or columns containing missing values, while fillna() will fill in the missing values with a specified value.

# Remove rows with any missing values
df.dropna()

# Remove columns with any missing values  
df.dropna(axis=1)  

# Fill missing values in column A with 0
df[‘A‘].fillna(0) 

# Forward fill missing values in column A
df[‘A‘].fillna(method=‘ffill‘)

Use dropna() cautiously as it can significantly reduce the data size. fillna() offers more flexibility, with options to fill with a scalar value, forward or backward fill from adjacent rows, or interpolate missing numbers.

8. duplicated() / drop_duplicates()

duplicated() detects duplicate rows, returning a boolean mask. drop_duplicates() removes those duplicate rows.

# Identify duplicate rows
df.duplicated()

# Remove duplicate rows, keeping first occurrence 
df.drop_duplicates()

# Remove duplicate rows, keeping last occurrence
df.drop_duplicates(keep=‘last‘)  

Removing duplicate data is another important data cleaning task during EDA.

9. sort_values()

To sort a DataFrame by one or more columns, use sort_values(). You can specify ascending or descending order.

# Sort by column A in ascending order
df.sort_values(‘A‘) 

# Sort by column A in descending order
df.sort_values(‘A‘, ascending=False)

# Sort by column A then column B, both ascending  
df.sort_values([‘A‘, ‘B‘])

Sorting can help spot outliers or invalid values, like dates in the future or negative ages.

10. groupby()

groupby() allows you to split a DataFrame into groups based on one or more columns, apply a function to each group, and combine the results.

# Calculate mean of column A for each value in column D
df.groupby(‘D‘)[‘A‘].mean()

# Calculate number of rows and mean of columns A, B and C for each value in D
df.groupby(‘D‘).agg({‘A‘: ‘mean‘, ‘B‘: ‘mean‘, ‘C‘: ‘size‘})  

This is incredibly useful for comparing metrics across categories or segments.

11. pivot_table()

pivot_table() creates a spreadsheet-style pivot table as a DataFrame. It can take multiple columns or indexes, aggregate by different functions, and handle missing data.

# Pivot table of mean A and B, indexed by D and E 
pd.pivot_table(df, values=[‘A‘, ‘B‘], index=[‘D‘, ‘E‘], aggfunc=‘mean‘)

This allows multidimensional summarization and comparison of metrics.

12. melt()

melt() is useful for transforming wide-format data into long-format. It takes columns and "melts" them into rows, creating a new DataFrame with two columns: variable and value.

# Melt columns A, B, C into rows  
pd.melt(df, id_vars=[‘D‘], value_vars=[‘A‘, ‘B‘, ‘C‘])

This can make data easier to plot or feed into machine learning algorithms that expect features in rows.

13. merge()

merge() allows you to combine DataFrames based on common columns or indexes, similar to SQL joins. You can perform inner, outer, left and right merges.

# Inner join DataFrames df1 and df2 on column D
df1.merge(df2, on=‘D‘)

# Left join DataFrames df1 and df2 on column D
df1.merge(df2, on=‘D‘, how=‘left‘)  

Merging is essential when data is spread across multiple files or tables that need to be combined for analysis.

14. concat()

To stack multiple DataFrames vertically or horizontally, use concat().

# Stack DataFrames vertically
pd.concat([df1, df2]) 

# Stack DataFrames horizontally  
pd.concat([df1, df2], axis=1)

This is helpful for combining data from multiple sources into one DataFrame.

15. map()

map() allows you to apply a function or mapping to each element of a Series.

# Create mapping of values 
mapping = {‘red‘: 0, ‘green‘: 1, ‘blue‘: 2}

# Map values in column D using mapping
df[‘D‘] = df[‘D‘].map(mapping)  

This is often used to recode categorical variables to numeric or vice versa.

16. apply()

For more complex element-wise transformations, use apply(). It applies a function along a DataFrame axis.

# Apply custom function to each row
df.apply(custom_function, axis=1) 

# Apply lambda function to column A
df[‘A‘].apply(lambda x: x**2) 

This allows for very flexible data transformations during EDA and feature engineering.

17. cut() / qcut()

cut() and qcut() bin continuous numeric data into discrete intervals. cut() creates equal-sized bins, while qcut() creates quantile bins with equal number of observations.

# Bin column A into 5 equal-sized intervals  
pd.cut(df[‘A‘], bins=5) 

# Bin column A into quantiles  
pd.qcut(df[‘A‘], q=4)

Binning is useful for understanding the distribution of numeric variables and can help in spotting relationships with other variables.

18. get_dummies()

get_dummies() converts categorical variables into dummy/indicator variables. It creates new binary columns for each category.

pd.get_dummies(df[‘D‘])

Output:

   blue  green  red
0     0      0    1
1     0      1    0
2     1      0    0

Dummy coding is often required before applying machine learning algorithms that can‘t handle categorical data directly.

19. corr()

To calculate pairwise correlation between columns, use corr(). It returns a correlation matrix as a DataFrame.

df.corr()  

Output:

          A         B         C
A  1.000000  0.117510  0.871754
B  0.117510  1.000000  0.120390
C  0.871754  0.120390  1.000000

Correlation gives an initial indication of relationships between numeric features, helping to spot important predictors, collinearity, or variables that can be dropped.

20. plot()

Finally, pandas provides a built-in plot() function for quickly creating basic visualizations directly from DataFrames or Series.

# Histogram of column A
df[‘A‘].plot(kind=‘hist‘)

# Scatter plot of column A vs B  
df.plot(x=‘A‘, y=‘B‘, kind=‘scatter‘)

# Boxplot of column A grouped by D
df.boxplot(column=‘A‘, by=‘D‘)

Visualization is a key component of EDA for understanding distributions, spotting patterns and outliers, and communicating insights.

Conclusion

Exploratory data analysis is a critical first step in any data science project, and pandas provides a powerful toolkit for quickly and efficiently exploring data in Python. The 20 functions covered here – head/tail, info, describe, value_counts, unique/nunique, isnull/notnull, dropna, duplicated, sort_values, groupby, pivot_table, melt, merge, concat, map, apply, cut/qcut, get_dummies, corr, and plot – are essential tools for inspecting, cleaning, transforming, and visualizing data during EDA.

By leveraging these pandas functions, you can significantly speed up the EDA process, generate valuable insights about the data, and identify potential issues or areas for further investigation. Effective EDA lays the foundation for all subsequent analysis, feature engineering, and modeling, so it‘s worth investing the time to learn and apply these core pandas capabilities.

Of course, these 20 functions only scratch the surface of pandas‘ full functionality. To further expand your pandas and EDA skills, check out the official pandas documentation, which provides detailed explanations and examples of all the library‘s features. As you work on more data science projects and encounter new data challenges, continue exploring pandas and adding more tools to your EDA toolbox.

Happy data exploring with pandas!