The Ultimate Guide to Pandas for Data Science
Introduction
Data is the new oil powering the engines of the digital economy. The global datasphere is predicted to grow to 175 zettabytes by 2025, a compounded annual growth rate of 61% [1]. As the volume, variety and velocity of data continues to explode, data scientists need powerful tools to efficiently process, manipulate and extract insights from big data. This is where pandas, the open source Python Data Analysis Library, comes in.
Since its creation by Wes McKinney in 2008, pandas has become an indispensable tool for data wrangling, analysis and visualization among data scientists, analysts and researchers. It is the 5th most used Python package with over 17M downloads per month [2]. Pandas‘ powerful, flexible and easy-to-use data structures like Series and DataFrame make it the go-to library for working with tabular, matrix-like datasets. Understanding how to leverage pandas effectively is now a must-have skill for any practicing data scientist.
In this ultimate guide, we‘ll take a deep dive into the pandas library from a data science and machine learning perspective. We‘ll cover its key features, core data structures, essential functions and real-world use cases with detailed code examples. Whether you‘re a pandas beginner or an experienced user looking to level up, this guide has you covered. Let‘s get started!
Why Pandas for Data Science?
Pandas offers several compelling benefits for data science:
- Intuitive data structures for data manipulation, alignment and analysis
- Fast, flexible and expressive tools for working with structured (tabular) and time series data
- Intelligent data alignment and integrated handling of missing values
- Flexible reshaping and pivoting of datasets
- Slicing, indexing and subsetting large datasets based on labels
- Merging and joining datasets
- Robust IO tools for loading and saving data across different formats
- Time series functionality: date range generation, frequency conversion, moving window statistics
- Powerful group by functionality for aggregating and transforming datasets
- Seamless integration with the PyData stack including NumPy, SciPy and Matplotlib
Installing and Importing Pandas
Pandas can be installed easily using pip:
pip install pandasTo use pandas in your Python code, import it under the conventional alias pd:
import pandas as pdUnderstanding Pandas Data Structures
Pandas provides two primary data structures: Series and DataFrame.
Series
A Series is a one-dimensional labeled array that can hold data of any type (integer, string, float, Python objects, etc.). It can be thought of as a single column in a spreadsheet.
s = pd.Series([1, 3, 5, 7, 9])DataFrame
A DataFrame is a 2-dimensional labeled data structure with columns of potentially different types, like a spreadsheet or SQL table. You can think of it as a dict-like container for Series objects. It is generally the most commonly used pandas object.
data = {‘country‘: [‘Belgium‘, ‘India‘, ‘Brazil‘],
‘capital‘: [‘Brussels‘, ‘New Delhi‘, ‘Brasília‘],
‘population‘: [11190846, 1303171035, 207847528]}
df = pd.DataFrame(data)Loading Data into Pandas
Pandas supports reading data from a wide variety of sources including CSV, Excel, SQL, JSON and more. Here are some common data loading functions:
df = pd.read_csv(‘data.csv‘)
df = pd.read_excel(‘data.xlsx‘, sheet_name=‘Sheet1‘)
df = pd.read_json(‘data.json‘)
df = pd.read_sql(‘SELECT * FROM table‘, con)Inspecting DataFrames
Once you‘ve loaded a DataFrame, you‘ll typically want to inspect it to understand the structure, data types, summary statistics, etc. Key functions for DataFrame inspection include:
df.head() # first 5 rows
df.tail() # last 5 rows
df.info() # index, data types, memory usage
df.describe() # summary statistics
df.shape # number of rows and columnsSelecting and Filtering Data
Pandas provides rich capabilities for selecting and filtering data in a DataFrame using labels, positions, or boolean conditions.
Selecting Columns
To select a single column, use square brackets [] with the column name. This returns a Series.
df[‘country‘] # select single column
df[[‘country‘, ‘capital‘]] # select multiple columns Selecting Rows by Position
To select rows by integer position, use iloc.
df.iloc[1] # second row
df.iloc[1:4] # second to fourth rows
df.iloc[[0,2,4]] # first, third and fifth rowsSelecting Rows by Label
To select rows by label, use loc.
df.loc[1] # row with index label 1
df.loc[1:3] # rows with index labels 1 to 3 Boolean Indexing
To select rows based on a boolean condition:
df[df.population > 1000000000] # population greater than 1 billionData Cleaning and Preparation
Real-world data is messy and needs cleanup before analysis. Typical issues include missing values, duplicate data, inconsistent formatting, etc. Pandas provides functions to handle these issues.
Handling Missing Data
Missing values are represented as NaN in pandas. Key functions for detecting and handling missing values are:
df.isnull() # returns boolean mask indicating missing values
df.dropna() # drop rows with missing values
df.fillna(0) # fill missing values with 0Removing Duplicates
To remove duplicate rows:
df.drop_duplicates() # drop duplicate rowsReplacing Values
To replace values:
df.replace(-999, np.nan) # replace -999 with NaNRenaming Columns
To rename columns:
df.rename(columns={‘old_name‘: ‘new_name‘})Data Transformation
Data transformation is the process of modifying the format or structure of data to make it suitable for analysis. Key data transformation operations in pandas include:
Sorting
To sort a DataFrame by a column:
df.sort_values(‘country‘) # sort rows by countryGrouping and Aggregation
The groupby function splits a DataFrame into groups based on a column, applies a function to each group and combines the results.
df.groupby(‘country‘).mean() # mean of numeric columns by country Merging and Joining
Pandas can combine multiple DataFrames by merging or joining based on a common column.
merged = pd.merge(left, right, on=‘key‘) Reshaping
Pandas provides functions to reshape data between long and wide formats.
df.pivot(index=‘date‘, columns=‘city‘, values=‘price‘) # long to wide
df.melt(id_vars=[‘name‘], value_vars=[‘height‘, ‘weight‘]) # wide to longData Visualization
Pandas integrates with Matplotlib allowing you to easily create plots from DataFrames.
df.plot() # line plot
df.plot.scatter(x=‘height‘, y=‘weight‘) # scatter plot
df.plot.bar() # bar plot
df.plot.hist() # histogram
df.plot.box() # box plotDescriptive Statistics
Pandas provides functions to generate descriptive statistics of data.
df.describe() # summary statistics
df.mean() # mean of columns
df.corr() # correlation between columns
df.count() # count of values
df.min() # minimum value
df.max() # maximum valueTime Series Analysis
Pandas has strong support for time series data. DatetimeIndex is the core data structure for working with dates and times.
df.set_index(‘date‘) # set date column as index
df.resample(‘M‘).mean() # resample to monthly frequency and take mean
df.shift(1) # shift index by 1
df.rolling(window=30).mean() # rolling window averageCase Study: Analyzing COVID-19 Data with Pandas
To showcase pandas‘ capabilities, let‘s analyze a real-world COVID-19 dataset from Our World in Data [3]. The dataset has daily COVID-19 stats for countries worldwide. Our goal is to load this data into pandas, prep it for analysis, and visualize some interesting trends.
Load Data
First, we load the CSV file into a DataFrame.
df = pd.read_csv(‘owid-covid-data.csv‘)Inspect Data
Let‘s check the shape, data types and first few rows.
print(df.shape)
print(df.info())
print(df.head())Output:
(180140, 67)
<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 180140 entries, 0 to 180139
Data columns (total 67 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 iso_code 175600 non-null object
1 continent 175600 non-null object
2 location 180140 non-null object
3 date 180140 non-null object
4 total_cases 180140 non-null float64
5 new_cases 180140 non-null float64
...
iso_code continent location date total_cases new_cases ...
0 AFG Asia Afghanistan 2020-02-24 5.0 5.0
1 AFG Asia Afghanistan 2020-02-25 5.0 0.0
2 AFG Asia Afghanistan 2020-02-26 5.0 0.0
3 AFG Asia Afghanistan 2020-02-27 5.0 0.0
4 AFG Asia Afghanistan 2020-02-28 5.0 0.0The dataset has 180K rows and 67 columns. It contains data for multiple countries (location column) for each date.
Data Preparation
Let‘s select columns of interest and replace nulls.
subset = df[[‘location‘, ‘date‘, ‘total_cases‘, ‘new_cases‘, ‘total_deaths‘, ‘new_deaths‘,
‘total_vaccinations‘, ‘people_vaccinated‘, ‘population‘]]
subset = subset.fillna(0) Let‘s convert date to datetime and set it as index.
subset[‘date‘] = pd.to_datetime(subset[‘date‘])
subset.set_index(‘date‘, inplace=True)Analyzing Global Trends
Let‘s group by date and sum to get worldwide daily stats.
global_stats = subset.groupby(‘date‘).sum()
print(global_stats.tail())Output:
total_cases new_cases total_deaths new_deaths total_vaccinations people_vaccinated population
date
2023-05-17 769574014.0 240577.0 6925241.0 1269.0 13436509270.0 5983765067.0 7.96e+09
2023-05-18 770070850.0 496836.0 6927335.0 2094.0 13457167941.0 5991330830.0 7.96e+09
2023-05-19 770501939.0 431089.0 6929172.0 1837.0 13478043200.0 5999014933.0 7.96e+09
2023-05-20 770793146.0 291207.0 6930417.0 1245.0 13492959805.0 6004279021.0 7.96e+09
2023-05-21 770793146.0 0.0 6930417.0 0.0 13508158974.0 6009597644.0 7.96e+09This shows the latest worldwide COVID-19 stats. As of May 2021, there have been over 770M cases, 6.9M deaths and 13.5B vaccinations globally.
Let‘s visualize new cases and deaths over time.
global_stats[[‘new_cases‘, ‘new_deaths‘]].plot(figsize=(12,6))
This shows the waves of COVID-19 globally, with major spikes in late 2020 and mid 2021 driven by the emergence of new variants.
Comparing Countries
Let‘s compare key metrics across G7 countries (Canada, France, Germany, Italy, Japan, UK, USA).
g7_countries = [‘Canada‘, ‘France‘, ‘Germany‘, ‘Italy‘, ‘Japan‘, ‘United Kingdom‘, ‘United States‘]
g7_stats = subset[subset.location.isin(g7_countries)]
g7_totals = g7_stats.groupby(‘location‘).max()
g7_totals[‘cases_per_million‘] = g7_totals.total_cases * 1e6 / g7_totals.population
g7_totals[‘deaths_per_million‘] = g7_totals.total_deaths * 1e6 / g7_totals.population
print(g7_totals[[‘location‘, ‘cases_per_million‘, ‘deaths_per_million‘, ‘people_vaccinated‘]])Output:
cases_per_million deaths_per_million people_vaccinated
location
Canada 104738.0 1138.0 38649815.0
France 348828.0 2404.0 55199353.0
Germany 223261.0 1839.0 64521906.0
Italy 310921.0 3116.0 52645820.0
Japan 74524.0 490.0 104956917.0
United Kingdom 283321.0 2769.0 53598133.0
United States 304024.0 3222.0 232894395.0This compares total cases, deaths and vaccinations per capita across G7 nations. France has the highest case rate, while the US has the highest death rate. Japan has the highest number of vaccinations.
We can visualize vaccination progress:
ax = g7_totals.plot.bar(y=‘people_vaccinated‘, figsize=(12,6), fontsize=13)
ax.set_xlabel("Country")
ax.set_ylabel("People Vaccinated")
This shows the US leads in total vaccinations, followed by Japan and Germany.
Conclusion
This guide provided a comprehensive overview of the pandas library and its applications in data science. We covered pandas‘ key features, core data structures, essential functions and real-world use cases including data loading, inspection, cleaning, transformation, visualization and analysis.
The COVID-19 case study demonstrated how we can use pandas to gain valuable insights from real-world data. We loaded data into a DataFrame, prepared it for analysis, calculated global trends and compared key metrics across countries, all with a few lines of pandas code.
Pandas is an incredibly powerful and flexible tool for data wrangling and analysis. Mastering it will significantly boost your efficiency and effectiveness as a data scientist. Keep practicing with pandas on a variety of datasets and integrate it into your data science workflow. The more you use it, the more you‘ll appreciate its capabilities.
References
[1] IDC Global Dataspherehttps://www.idc.com/getdoc.jsp?containerId=IDC_P38353 [2] Most Popular Python Packages
https://hugovk.github.io/top-pypi-packages/ [3] Our World in Data COVID-19 Dataset
https://github.com/owid/covid-19-data
