Data Science with Pandas: 2 Minute Guide to Key Concepts

Pandas is the most popular and widely used open source library for data manipulation and analysis in Python. It provides high-performance, easy-to-use data structures and tools for working with structured data. According to the 2021 Stack Overflow Developer Survey, pandas is the 5th most commonly used library/framework overall, and the most popular data science library. On GitHub, the pandas repository has over 34,000 stars and 1,800 contributors, reflecting its broad adoption and active development.

What makes pandas so powerful and ubiquitous in the data science world? At its core, pandas provides two main data structures: Series (1-dimensional) and DataFrame (2-dimensional), which can handle a variety of data types (e.g. numeric, string, boolean, datetime). These data structures are built on top of NumPy arrays, which enables pandas to leverage high-performance vectorized operations. However, pandas adds many useful features on top of NumPy, such as:

• Labeled rows and columns for easier data access and alignment
• Handling of missing data
• Merging and joining datasets
• Reshaping and pivoting data
• Time series functionality
• Plotting functions

Let‘s dive into the key concepts and techniques that every data scientist using pandas should know.

Creating and Inspecting DataFrames

The fundamental data structure in pandas is the DataFrame, which you can think of as a relational table or a spreadsheet. DataFrames are designed to make working with tabular and mixed-type data intuitive and natural. To create a DataFrame, you can pass:

• A dictionary of lists, NumPy arrays, or Series
• A list of dicts or Series
• A Series
• Another DataFrame
• A NumPy structured or record array
• A dict of DataFrame objects

Here‘s an example of creating a DataFrame from a dictionary of lists:

import pandas as pd

data = {‘name‘: [‘Alice‘, ‘Bob‘, ‘Charlie‘, ‘David‘],
        ‘age‘: [25, 30, 35, 40],
        ‘salary‘: [50000, 60000, 70000, 80000]}

df = pd.DataFrame(data)
print(df)

This will output:

      name  age  salary
0    Alice   25   50000
1      Bob   30   60000
2  Charlie   35   70000
3    David   40   80000

After creating a DataFrame, you‘ll often want to inspect its contents and metadata. Some useful methods for this are:

• head(): Returns the first n rows (default 5)
• tail(): Returns the last n rows (default 5)
• info(): Prints a summary of the DataFrame, including column data types and non-null counts
• describe(): Generates descriptive statistics for numerical columns (count, mean, std, min, quartiles, max)
• shape: Prints the dimensions of the DataFrame (rows, columns)
• dtypes: Returns the data type of each column
• columns: Returns the column labels

For example:

print(df.head(2))
print(df.info())
print(df.describe())

Selecting and Filtering Data

One of the most common tasks in data analysis is selecting a subset of data based on some criteria. Pandas provides several ways to select and filter data in a DataFrame.

To select one or more columns, you can use:

• Square bracket notation: df[‘column_name‘] or df[[‘col1‘, ‘col2‘]]
• Dot notation: df.column_name (only for single columns)

To select rows by label or position, you can use:

• loc: Label-based indexing, e.g. df.loc[2] selects row with label 2
• iloc: Integer-based indexing, e.g. df.iloc[2] selects 3rd row
• []: Can select by label or position, e.g. df[2:4] selects 3rd and 4th rows

To filter rows based on a condition, you can use boolean indexing:

print(df[df[‘age‘] > 30])  # Select rows where age is greater than 30
print(df[(df[‘age‘] > 30) & (df[‘salary‘] > 60000)])  # Combining multiple conditions with & (and) or | (or)

Handling Missing Data

Real-world datasets often contain missing or null values, which can cause issues for data analysis and modeling. Pandas represents missing values as NaN (Not a Number) and provides functions to detect, remove, and fill missing data.

To check for missing values, use:

• isnull(): Returns boolean mask indicating missing values
• notnull(): Returns boolean mask indicating non-missing values

To remove rows or columns containing missing values, use:

• dropna(): Removes rows or columns with any missing values
  • axis parameter specifies whether to drop rows (0) or columns (1)
  • how parameter specifies whether to drop if ‘any‘ or ‘all‘ values are missing
  • subset parameter allows dropping only if certain columns have missing values

To fill in missing values, use:

• fillna(): Fills missing values with specified value or method
  • Scalar value: e.g. df.fillna(0) fills all missing values with 0
  • method parameter: ‘ffill‘ (forward fill) or ‘bfill‘ (backward fill)
  • Dictionary: Specify different fill values for each column

Here‘s an example showing some of these methods:

# Create DataFrame with missing values
df_missing = pd.DataFrame({‘A‘:[1,2,np.nan,4], 
                           ‘B‘:[5,np.nan,np.nan,8], 
                           ‘C‘:[10,20,30,40]})

print(df_missing.isnull())
print(df_missing.dropna())
print(df_missing.fillna(method=‘ffill‘))  

Merging and Joining DataFrames

Pandas provides functions to combine multiple DataFrames based on a common column or index. This is useful for bringing in additional data to augment your analysis.

The main methods for merging and joining are:

• concat(): Concatenates DataFrames vertically (adding rows) or horizontally (adding columns)

  • axis parameter specifies vertical (0, default) or horizontal (1) concatenation
  • join and join_axes allow specifying which indexes to use for joining

• merge(): Merges DataFrames by performing a database-style join on one or more common columns

  • how parameter specifies type of merge: ‘inner‘ (default), ‘outer‘, ‘left‘, ‘right‘
  • on parameter specifies column(s) to join on
  • left_on and right_on specify join columns separately for left and right DataFrames
  • left_index and right_index allow merging on indexes instead of columns

• join(): Joins columns of another DataFrame by matching indexes

  • Can pass a list of DataFrames to join with
  • how parameter specifies type of join: ‘left‘ (default), ‘right‘, ‘inner‘, ‘outer‘

Here‘s an example demonstrating a left join:

# Create DataFrames
df1 = pd.DataFrame({‘key‘: [‘A‘, ‘B‘, ‘C‘, ‘D‘],
                    ‘value1‘: [1, 2, 3, 4]})
df2 = pd.DataFrame({‘key‘: [‘B‘, ‘D‘, ‘E‘, ‘F‘],
                    ‘value2‘: [5, 6, 7, 8]})

# Left join df1 with df2
merged = df1.merge(df2, how=‘left‘, on=‘key‘)
print(merged)

Output:

  key  value1  value2
0   A       1     NaN
1   B       2     5.0
2   C       3     NaN
3   D       4     6.0

Reshaping Data

Pandas has functions to transform and reshape DataFrames for different analyses and visualizations. The main methods are:

• pivot(): Reshapes data based on column values, going from long to wide format

  • index specifies column(s) to use as new index
  • columns specifies column to use as new columns
  • values specifies column(s) to use as new values

• melt(): Unpivots data from wide to long format

  • id_vars specifies columns to use as identifier variables
  • value_vars specifies columns to unpivot
  • var_name and value_name specify names of new variable and value columns

• stack(): Moves column index to row index, reshaping columns into rows

  • Increases number of rows and decreases number of columns

• unstack(): Moves row index to column index, reshaping rows into columns

  • Decreases number of rows and increases number of columns

Here‘s an example showing how to pivot data:

# Create DataFrame
df = pd.DataFrame({‘Name‘: [‘Alice‘, ‘Bob‘, ‘Charlie‘],
                   ‘Subject‘: [‘Math‘, ‘Math‘, ‘Science‘], 
                   ‘Score‘: [85, 92, 88]})

# Pivot data to show scores by name and subject
pivoted = df.pivot(index=‘Name‘, columns=‘Subject‘, values=‘Score‘)
print(pivoted)

Output:

Subject    Math  Science
Name                    
Alice        85      NaN
Bob          92      NaN
Charlie     NaN       88

Applying Functions

Pandas allows applying functions to DataFrames for data transformation and aggregation. The key methods are:

• apply(): Applies function along an axis (0 for rows, 1 for columns)

  • Passes Series to function for each column/row
  • Returns Series or DataFrame with results
  • Can specify raw=True to pass underlying NumPy array to function instead of Series

• applymap(): Applies function elementwise to each cell in DataFrame

  • Useful for transforming all values, e.g. converting to string, rounding decimals

• map(): Applies function elementwise to Series

  • Can map values according to input dictionary or Series
  • Useful for recoding categorical variables

Here‘s an example using apply() to calculate the range for each column:

df = pd.DataFrame(np.random.randint(0, 100, size=(5, 3)), columns=[‘A‘, ‘B‘, ‘C‘])
print(df)

print(df.apply(lambda x: x.max() - x.min()))

Output:

    A   B   C
0  55  58  79
1  23  35  62
2   7  64  37
3  33  29  13
4  29  74  23

A    48
B    45
C    66
dtype: int64

Plotting Data

Pandas integrates with Matplotlib to enable directly plotting from DataFrames using the plot() method. This allows quick exploratory visualizations without having to write full Matplotlib code. Some common plots include:

• Line plot: df.plot() (default)
• Bar plot: df.plot(kind=‘bar‘)
• Horizontal bar: df.plot(kind=‘barh‘)
• Histogram: df.plot(kind=‘hist‘)
• Box plot: df.plot(kind=‘box‘)
• Area plot: df.plot(kind=‘area‘)
• Scatter plot: df.plot(x=‘column1‘, y=‘column2‘, kind=‘scatter‘)

The plotting API is quite customizable, allowing specifying labels, titles, legends, colors, styles, subplots, and more. See the pandas plotting documentation for more details and examples.

Performance Tips

While pandas is highly efficient and flexible, you may run into performance bottlenecks when working with very large datasets. Here are some tips for optimizing pandas code:

• Use vectorization instead of loops: Pandas is built on top of NumPy, so using vectorized operations instead of iterating over rows can result in orders of magnitude speed improvements.

• Specify data types when reading data: Pandas will try to infer data types automatically, but you can set them explicitly using the dtype parameter in read_csv() and other I/O functions to reduce memory usage and improve performance.

• Use efficient file formats: Pandas supports reading and writing data in various formats, including CSV, Excel, SQL, JSON, HDF5, and Parquet. Using binary formats like HDF5 and Parquet can significantly speed up I/O and reduce file sizes.

• Choose appropriate algorithms and data structures: Different methods for grouping, joining, sorting, etc. have different time and space complexities. See the pandas performance tips for guidance.

• Persist data in memory: If you‘re repeatedly accessing the same data, it may be more efficient to load it into memory first using methods like to_numpy(), to_dict(), or to_records() instead of reading from disk each time.

• Use other libraries for big data: Pandas is not designed to handle datasets larger than memory. For truly massive datasets, consider using libraries like Dask, Vaex, or Modin that can process data in parallel and out-of-core.

Conclusion

Pandas is a powerful and indispensable library for data science in Python. In this guide, we‘ve covered the key concepts and techniques for working with data in pandas, including:

• Creating and inspecting DataFrames
• Selecting and filtering data
• Handling missing values
• Merging and joining datasets
• Reshaping data
• Applying functions
• Plotting data
• Optimizing performance

However, we‘ve only scratched the surface of what pandas can do. To dive deeper, I recommend the following resources:

• Official pandas documentation
• 10 Minutes to pandas
• Python for Data Analysis by Wes McKinney, creator of pandas
• Effective Pandas by Matt Harrison

Remember, the best way to learn pandas is through practice and exploration. Don‘t be afraid to experiment, make mistakes, and consult the documentation and community for help. With pandas in your data science toolkit, you‘ll be well-equipped to tackle a wide range of data challenges.