13 Essential Pandas Functions Every Data Scientist Should Know

If you‘re working with data in Python, chances are you‘re using the powerful pandas library. Pandas provides easy-to-use data structures and functions that make data manipulation, analysis and cleaning a breeze. It‘s an indispensable tool in the data scientist‘s toolkit.

While pandas has a vast array of capabilities, there are some key functions that you‘ll use over and over again. Mastering these core pandas operations will make you a more effective and efficient data wrangler.

In this article, we‘ll dive deep into the 13 most important pandas functions every data scientist should know, complete with code examples and use cases. Consider this your handy reference guide to the pandas essentials.

1. Reading and Writing Data with read_csv() and to_csv()

The first step in any data project is acquiring your data. More often than not, you‘ll be working with data stored in CSV (comma-separated values) files. Pandas makes it incredibly easy to read CSV data into a DataFrame with the read_csv() function.

For example, let‘s say we have a file called "data.csv" containing customer information:

import pandas as pd

df = pd.read_csv(‘data.csv‘)

Just like that, we‘ve read our CSV data into the DataFrame df and we‘re ready to start exploring and analyzing it.

When you‘re done working with your data, you can use the to_csv() function to write the DataFrame back out to a CSV file:

df.to_csv(‘output.csv‘, index=False)  

Here we specify index=False so row indexes are not included in the output file.

2. Previewing Your Data with head() and tail()

After loading data into a DataFrame, you‘ll usually want to take a quick peek at it to get a sense of what you‘re working with. That‘s where the handy head() and tail() functions come in.

head(n) returns the first n rows of the DataFrame, while tail(n) returns the last n rows. By default, they return 5 rows if you don‘t specify n.

df.head()  # preview first 5 rows
df.tail(10)  # preview last 10 rows  

These functions are great for quickly spotting any data quality issues or outliers before digging deeper. Think of them as a way to "look before you leap" into your data.

3. Generating Summary Statistics with describe()

Once you‘ve eyeballed your data, the next step is usually to compute some summary statistics to better understand the distribution and central tendency of your numeric columns. The describe() function does this in one fell swoop.

df.describe()

This will output a table containing the count, mean, standard deviation, minimum, 25th percentile, median, 75th percentile, and maximum for each numeric column. It‘s a great way to quickly spot any outliers or data skew.

You can also use describe() on non-numeric columns to get the count, unique values, top value (mode), and frequency of the top value:

df.describe(include=‘object‘)  

4. Checking Data Types and Memory Usage with info() and dtypes

Data comes in many different types – integers, floats, strings, datetimes, booleans, objects, and more. It‘s important to understand what data types you‘re working with, as this determines what operations you can perform on each column.

You can check the data type of each column in a DataFrame using the dtypes attribute:

df.dtypes

For a more detailed overview, use the info() function:

df.info()

This outputs the number of rows, number of columns, column data types, and memory usage for the DataFrame. It‘s a handy way to quickly check if your data was loaded correctly and identify any columns that may need type conversion (more on this next).

5. Converting Data Types with astype()

Sometimes you‘ll load data into a DataFrame and find that some columns were parsed as the wrong data type. For example, maybe a numeric column got parsed as strings. In this case, you‘ll need to convert or "cast" the column to the correct data type using the astype() function.

Let‘s say we have a DataFrame with a "price" column that was loaded as strings but should be floats:

df[‘price‘] = df[‘price‘].astype(‘float‘)

You can convert to any valid data type, such as int, float, string, datetime, and more.

Converting to the right data type is crucial because it enables you to properly use pandas‘ built-in functions on your data. For example, you can‘t calculate the mean of a string column.

6. Accessing Subsets of Data with loc[] and iloc[]

One of the most powerful features of pandas is the ability to quickly access subsets of your data using label-based (loc[]) or integer-based (iloc[]) indexing. This lets you select specific rows and columns from a DataFrame by their labels or integer positions.

For example, to select rows 0-2 and columns "name" and "age":

subset = df.loc[0:2, [‘name‘, ‘age‘]]

loc[] is inclusive of the end index, so 0:2 selects the first 3 rows.

To select the same subset by integer position:

subset = df.iloc[0:3, [0,2]]  

iloc[] is exclusive of the end index, so 0:3 selects rows 0, 1, and 2. Column integers refer to their position from left to right.

Indexing is one of the most used pandas operations and is essential for filtering, selecting, and reshaping your data. It‘s worth practicing until it becomes second nature.

7. Working with Dates and Times using to_datetime()

Pandas has robust support for working with dates, times, and time series data. However, these values are often loaded as strings initially. To unlock pandas‘ date/time capabilities, you first need to convert them to datetime data type using to_datetime().

For example, to convert a column of string dates to datetime:

df[‘date‘] = pd.to_datetime(df[‘date‘])  

Once converted to datetime, you can do things like extract components of the date (year, month, day, etc.), calculate time differences, resample time series, and much more. Pandas datetime tools are indispensable for any time series analysis.

8. Counting Unique Values with value_counts()

A common task in data analysis is counting the number of unique values in a column and calculating the frequency of each unique value. The value_counts() function does just that.

counts = df[‘category‘].value_counts()

This returns a Series containing the unique values in the "category" column as the index and the count of each value as the Series values, sorted in descending order by default.

value_counts() is useful for understanding the distribution of categorical variables, finding the most common values, and checking for data imbalance (e.g. if one category is far more frequent than others). You can also quickly plot a bar chart or pie chart of the value counts for a visual representation.

9. Removing Duplicate Data with drop_duplicates()

Duplicate data can sneak into your datasets for many reasons – maybe a data entry error, a join gone wrong, or data collected from multiple sources. Regardless of the cause, you usually want to remove duplicate rows before analyzing your data to avoid skewed or incorrect results.

Pandas makes de-duplication easy with the drop_duplicates() function:

df = df.drop_duplicates()

By default, this will remove any rows that are exact duplicates across all columns. You can also specify a subset of columns to check for duplication, or keep the first/last occurrence of each duplicate group.

Dropping duplicates is a key data cleaning step that you‘ll want to perform early on in your data processing pipeline.

10. Splitting Data into Groups with groupby()

Many data analysis tasks involve splitting data into groups based on some criteria, applying a function to each group, and combining the results. This is known as the "split-apply-combine" pattern, and pandas implements it with the groupby() function.

To use groupby(), you specify one or more columns to group by and apply an aggregation function:

grouped = df.groupby(‘category‘)[‘price‘].mean()

This will group the DataFrame by the "category" column, select the "price" column, and calculate the mean price for each category.

You can apply any function to the groups, such as count, sum, min, max, median, standard deviation, and more. groupby() is incredibly powerful and is often used in conjunction with other functions like agg() and transform() for more complex grouped operations.

11. Combining Data with merge()

In the real world, data often comes in multiple tables that need to be combined based on a common key. This is where the merge() function shines.

merge() lets you combine two DataFrames based on a shared column, similar to a SQL join. You specify the column(s) to join on and the type of join to perform (inner, outer, left, or right).

For example, let‘s say we have two DataFrames: customers and orders. We can combine them based on a shared "customer_id" column:

merged = pd.merge(customers, orders, on=‘customer_id‘)

By default, this performs an inner join, retaining only rows that have matching customer_id values in both tables. You can specify other join types using the how parameter.

merge() is a crucial tool for combining disparate data sources into a unified dataset ready for analysis. It‘s also very flexible, allowing for more complex joins involving multiple keys.

12. Sorting Data with sort_values() and sort_index()

Sorting is a fundamental data operation that you‘ll use frequently in your analyses. Pandas provides two main functions for sorting: sort_values() for sorting by column values, and sort_index() for sorting by row or column index.

To sort a DataFrame by one or more columns:

df = df.sort_values([‘category‘, ‘price‘], ascending=[True, False])

This will sort the DataFrame first by "category" in ascending order, then by "price" in descending order.

To sort by index:

df = df.sort_index()

By default, this sorts the row index in ascending order. You can specify axis=1 to sort by column index instead.

Sorting is often used to arrange data in a meaningful order, find the top or bottom N values, or prepare data for further processing. It‘s a simple but essential operation to have in your pandas toolkit.

13. Handling Missing Data with fillna()

Missing data is a fact of life in data science. Whether due to data entry issues, data corruption, or data that simply doesn‘t exist, missing values can cause issues with your analysis if not handled properly.

Pandas represents missing data with the special NaN (Not a Number) value. You can check for NaN values using the isna() function, and count them with isna().sum().

One common way to deal with missing data is to fill the NaN values with some other value, such as zero, the mean of the column, or the last valid value. This is where the fillna() function comes in handy.

For example, to fill all NaN values in a DataFrame with zero:

df = df.fillna(0)

You can specify different fill values for each column using a dictionary:

df = df.fillna({‘price‘: 0, ‘category‘: ‘Unknown‘}) 

fillna() provides a quick way to replace missing data with sensible defaults. Of course, you should always think carefully about what fill values make sense for your specific data and analysis goals. Sometimes it may be better to simply remove rows with missing data using dropna() instead.

Putting It All Together

We‘ve covered a lot of ground in this article, from reading in data to cleaning, filtering, transforming, combining, and analyzing it. While we‘ve only scratched the surface of what you can do with pandas, mastering these 13 essential functions will put you well on your way to data science success.

Of course, the best way to cement your pandas knowledge is through hands-on practice. Try loading some real-world datasets and applying the functions we‘ve covered here. Experiment, explore, and don‘t be afraid to consult the pandas documentation when you get stuck.

With time and practice, manipulating data with pandas will start to feel like second nature. And once you‘re comfortable with these core functions, you‘ll be ready to dive into some of pandas‘ more advanced capabilities, such as time series analysis, data reshaping, and merging on indexes.

So fire up a Jupyter notebook, load some data, and start wrangling! The exciting world of data science awaits.