30 Essential Pandas Functions for Data Analysis & Manipulation

When it comes to data analysis and manipulation in Python, the pandas library is the go-to tool for data scientists and analysts. Pandas provides powerful, flexible and easy-to-use data structures and functions that make working with structured data a breeze.

In this comprehensive guide, we‘ll dive into 30 essential pandas functions that every data professional should know. Whether you‘re a pandas beginner or a seasoned expert, you‘ll find valuable tips, examples and best practices to take your data analysis skills to the next level. Let‘s get started!

Contents

• Introduction to Pandas
• Key Data Structures: Series & DataFrame
• Reading Data
• Inspecting Data
• Selecting Subsets of Data
• Manipulating Data
• Summarizing Data
• Handling Missing Data
• Combining Data
• Reshaping Data
• Working with Strings
• Working with Dates & Times
• Plotting Data
• Writing Data
• Conclusion

Introduction to Pandas

Pandas is an open-source Python package that provides fast, flexible, and expressive data structures designed to make working with structured (tabular, multidimensional, potentially heterogeneous) and time series data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python.

At the core of pandas are two primary data structures:

1. Series: one-dimensional labeled array capable of holding any data type
2. DataFrame: two-dimensional labeled data structure with columns of potentially different types

With these data structures and pandas‘ extensive set of functions, you can easily load, manipulate, analyze, model and visualize your data.

Key Data Structures: Series & DataFrame

Before diving into pandas functions, let‘s take a closer look at its two key data structures: Series and DataFrame.

A Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). You can think of it like a single column in a spreadsheet.

A DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table. The DataFrame consists of three principal components: the data, the index, and the columns.

Here‘s how you can create a Series and DataFrame:

import pandas as pd

# Create a Series
s = pd.Series([1, 3, 5, np.nan, 6, 8])

# Create a DataFrame
dates = pd.date_range("20130101", periods=6)  
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list("ABCD"))

Now that we understand pandas‘ data structures, let‘s explore some of the most useful functions for working with them.

Reading Data

The first step in data analysis is often getting your data into pandas. Pandas provides several functions for reading data from various sources:

• read_csv: Read a comma-separated values (CSV) file into DataFrame
• read_excel: Read an Excel file into a DataFrame
• read_json: Convert a JSON string to pandas object
• read_html: Read HTML tables into a list of DataFrame objects
• read_sql: Read SQL query or database table into a DataFrame

Here‘s an example of reading a CSV file:

df = pd.read_csv("data.csv")

Inspecting Data

After loading your data into a DataFrame, you‘ll want to explore it to understand its structure and content. Pandas provides many useful functions for inspecting your data:

• head(): Return the first n rows (default 5)
• tail(): Return the last n rows (default 5)
• info(): Print a concise summary of a DataFrame
• describe(): Generate descriptive statistics of DataFrame columns
• shape: Return a tuple representing the dimensionality of the DataFrame
• dtypes: Return the dtypes in the DataFrame
• sample(): Return a random sample of items from an axis of object

For example, to see the first 5 rows of a DataFrame:

df.head()

Selecting Subsets of Data

One of the most common tasks in data analysis is selecting subsets of data based on some criteria. Pandas provides powerful indexing and selection capabilities through these key functions:

• []: Select a single column or subset of rows and columns from a DataFrame
• loc[]: Access a group of rows and columns by label(s)
• iloc[] Access a group of rows and columns by integer position(s)
• query(): Query the columns of a DataFrame with a boolean expression
• isin(): Whether each element in the DataFrame is contained in values

For example, to select rows where the ‘A‘ column is greater than 0:

df[df.A > 0]  

Manipulating Data

After selecting your data, you‘ll often need to manipulate it in some way. Pandas has a wide range of functions for modifying and operating on data:

• drop(): Remove rows or columns by specifying label names and axis
• rename(): Alter axes labels
• sort_values(): Sort by the values along either axis
• nlargest()/nsmallest(): Return the first n rows ordered by a column
• where()/mask(): Replace values where the condition is False/True
• duplicated(): Return boolean Series denoting duplicate rows
• drop_duplicates(): Return DataFrame with duplicate rows removed

For instance, to drop a column from a DataFrame:

df = df.drop(columns=["B"])

Summarizing Data

Getting summary statistics and aggregations is an essential part of data analysis. Pandas offers many useful functions for summarizing data:

• unique(): Return unique values of Series object
• value_counts(): Return a Series containing counts of unique values
• sum(): Return the sum of the values over the requested axis
• mean(): Return the mean of the values over the requested axis
• min()/max(): Return the minimum/maximum of the values over the requested axis
• groupby(): Group DataFrame using a mapper or by a Series of columns
• agg(): Aggregate using callable, string, dict, or list of string/callables

For example, to get the mean of each numeric column:

df.mean()  

Handling Missing Data

Real-world data is messy and often contains missing values. Pandas provides useful functions for detecting and handling missing data:

• isna()/isnull(): Detect missing values
• notna()/notnull(): Detect non-missing values
• dropna(): Return object with labels on given axis omitted where alternately any or all of the data are missing
• fillna(): Fill NA/NaN values using the specified method

To drop rows with any missing values:

df = df.dropna()

Combining Data

Data often comes in multiple pieces that need to be put together. Pandas has several powerful methods for merging, joining and concatenating data:

• concat(): Concatenate pandas objects along a particular axis
• merge(): Merge DataFrame or named Series objects with a database-style join
• join(): Join columns of another DataFrame

Here‘s an example of concatenating two DataFrames vertically:

df_concat = pd.concat([df1, df2])

Reshaping Data

Reshaping data is a common task, especially when preparing it for analysis or visualization. Pandas provides functions for pivoting and reshaping data:

• melt(): Unpivot a DataFrame from wide to long format
• pivot(): Reshape data (produce a "pivot" table) based on column values
• stack(): Stack the prescribed level(s) from columns to index
• unstack(): Pivot a level of the (necessarily hierarchical) index labels

To pivot a DataFrame:

df_pivot = df.pivot(index="date", columns="variable", values="value")

Working with Strings

Pandas has vectorized string functions that make it easy to operate on string data. Some key string methods include:

• lower()/upper(): Convert strings to lowercase/uppercase
• split(): Split strings around given separator/delimiter
• replace(): Replace occurrences of pattern/regex in the Series/Index
• contains(): Test if pattern or regex is contained within a string of a Series or Index

For instance, to convert a string column to lowercase:

df["name"] = df["name"].str.lower()

Working with Dates & Times

Pandas has extensive capabilities for working with time series data. Key functions for handling dates and times include:

• to_datetime(): Convert argument to datetime
• dt accessor: Access datetime properties through an accessor (e.g. dt.year, dt.month)

Here‘s an example of converting a string column to datetime:

df["date"] = pd.to_datetime(df["date"]) 

Plotting Data

Pandas integrates directly with Matplotlib, allowing you to easily create plots from DataFrames and Series. Some basic plotting functions:

• plot(): Plot data and return plot object(s)
• hist(): Make a histogram of the DataFrame‘s columns
• boxplot(): Make a box plot of the DataFrame columns
• scatter_matrix(): Create a matrix of scatter plots

To create a line plot of a DataFrame:

df.plot()  
plt.show()

Writing Data

After manipulating and analyzing your data, you‘ll often want to write it out to a file. Pandas provides functions for writing data to various formats:

• to_csv(): Write object to a comma-separated values (CSV) file
• to_excel(): Write object to an Excel sheet
• to_json(): Convert the object to a JSON string
• to_sql(): Write records stored in a DataFrame to a SQL database

For example, to write a DataFrame to CSV:

df.to_csv("output.csv", index=False) 

Conclusion

We‘ve covered 30 of the most essential pandas functions for data analysis and manipulation. Of course, there are many more useful functions in pandas, but mastering these will give you a solid foundation for working with data in Python.

Remember, the key to becoming proficient with pandas is practice. The more you work with real datasets and apply these functions, the more comfortable and efficient you‘ll become. Don‘t be afraid to experiment, consult the documentation, and ask the community for help when you get stuck.

Happy data wrangling with pandas!