Column Non-Null Count Dtype

If you‘re working with data in Python, chances are you‘re using the pandas library. Pandas is an incredibly powerful open-source library that provides easy-to-use data structures and analysis tools. It has become an essential tool for data scientists, analysts, and anyone else who works with data.

One of the most important steps in any data analysis project is data exploration. Before you start building models or creating visualizations, you need to thoroughly understand the contents, structure, and quality of your dataset. Fortunately, pandas provides a variety of functions that make it easy to explore and gain insights from your data.

In this article, we‘ll walk through 15 of the most useful pandas functions for data exploration. For each function, we‘ll explain what it does and provide examples of how to use it on a sample dataset. By the end, you‘ll have a solid foundation for exploring and preprocessing your own data with pandas.

1. head()

The first thing you‘ll usually want to do when working with a new dataset is take a quick look at the contents. The head() function allows you to preview the first n rows of a DataFrame (by default, n=5). This is useful for getting a sense of what columns are present and what the data looks like.

For example, let‘s say we have a DataFrame called sales_data with information about product sales:

sales_data.head(10)

This would return the first 10 rows of the DataFrame, allowing us to see examples of the product names, sales amounts, dates, and other columns that are present.

2. info()

To get a more comprehensive overview of a DataFrame, we can use the info() function. This provides useful summary information like:

• The number of rows and columns
• The data types of each column
• The number of non-null values in each column
• The amount of memory the DataFrame is using

sales_data.info()

Output:
<class ‘pandas.core.frame.DataFrame‘>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 5 columns):

---

0 product_id 1000 non-null int64
1 product_name 1000 non-null object
2 category 1000 non-null object
3 price 1000 non-null float64
4 sales 1000 non-null int64
dtypes: float64(1), int64(2), object(2)
memory usage: 39.2+ KB

This gives us a nice snapshot of the key properties of our DataFrame. We can see at a glance how many rows and columns there are, which columns contain strings vs numbers, if there are any missing values, and how much space the DataFrame takes up in memory.

3. describe()

For an even more detailed statistical summary of a DataFrame, we can use describe(). This generates common descriptive statistics for each numerical column, including:

• count: number of non-null values
• mean: average value
• std: standard deviation
• min: smallest value
• 25%/50%/75%: 25th, 50th and 75th percentile values
• max: largest value

sales_data.describe()

Output:
product_id price sales
count 1000.000000 1000.000000 1000.000000
mean 4999.500000 56.790000 433.360000
std 2887.019487 26.984946 247.342586
min 0.000000 10.990000 20.000000
25% 2499.750000 32.990000 258.000000
50% 4999.500000 56.990000 362.000000
75% 7499.250000 70.990000 500.000000
max 9999.000000 119.990000 1000.000000

This summary gives us a good sense of the distribution and range of values for the numeric columns in our data. We can see right away that there is a wide range of product prices and sales amounts. The percentiles also give us a quick look at where the bulk of the values lie – for example, 75% of products have a price below $70.99.

4. shape

If we just want to know the dimensions of a DataFrame, we can access the shape attribute. This returns a tuple with the number of rows followed by the number of columns:

sales_data.shape

Output: (1000, 5)

So in this case, our DataFrame has 1,000 rows and 5 columns. Knowing the rough size of your dataset is useful for planning your analysis approach.

5. columns and index

Two other useful attributes of a DataFrame are columns and index. These return the column labels and row index labels, respectively:

sales_data.columns

Output: Index([‘product_id‘, ‘product_name‘, ‘category‘, ‘price‘, ‘sales‘], dtype=‘object‘)

sales_data.index

Output: RangeIndex(start=0, stop=1000, step=1)

Here we can see that our DataFrame has columns called ‘product_id‘, ‘product_name‘, etc., and the rows are indexed by a default numeric RangeIndex. Being able to easily access and inspect the row/column labels comes in handy during data exploration and cleaning.

6. astype()

In some cases, the data types of our columns might not be what we expect or want them to be. For instance, maybe a column of numbers is being stored as strings instead of integers. We can use the astype() function to convert columns to different data types:

sales_data["product_id"] = sales_data["product_id"].astype(‘string‘)

This would convert the ‘product_id‘ column from integer to string type. Updating data types is often a necessary step before performing certain operations or analysis.

7. sort_values()

Sorting is a common task during data exploration. We can use sort_values() to sort a DataFrame by one or more columns. By default, it sorts in ascending order, but we can specify ascending=False to sort in descending order:

sales_data.sort_values(by=[‘category‘, ‘sales‘], ascending=[True, False])

This would sort the DataFrame first by the ‘category‘ column in ascending order, then by the ‘sales‘ column in descending order within each category. Sorting can help us quickly identify the top or bottom values for columns of interest.

8. value_counts()

Speaking of identifying top values, the value_counts() function is useful for counting the unique values in a column. It returns a Series with counts of each unique value, sorted in descending order by default:

sales_data[‘category‘].value_counts()

Output:
electronics 400
clothing 300
home goods 300
Name: category, dtype: int64

So in this example, we can see that our DataFrame contains 400 electronics products, 300 clothing products, and 300 home goods. This gives us a quick sense of the composition of categorical variables.

9. drop_duplicates()

Duplicate rows are a common type of dirty data that we need to deal with during preprocessing. The drop_duplicates() function removes duplicate rows from a DataFrame:

sales_data.drop_duplicates(inplace=True)

Setting inplace=True means the change is made directly to the original DataFrame. By default, all columns are used to identify duplicates, but you can also specify a subset of columns to check for duplication. Removing duplicates helps ensure the accuracy and integrity of your dataset.

10. Subsetting with [], loc, and iloc

One of pandas‘ most powerful features is the ability to subset data in various ways. There are three main methods for selecting subsets of a DataFrame:

• Square brackets [] let you select columns by name
• loc lets you select rows and columns by name
• iloc lets you select rows and columns by integer index

For example, we could select just the ‘product_name‘ and ‘sales‘ columns like this:

sales_data[[‘product_name‘, ‘sales‘]]

Or we could select all rows where ‘category‘ is ‘electronics‘ and only the ‘price‘ column like this:

sales_data.loc[sales_data[‘category‘] == ‘electronics‘, ‘price‘]

Finally, we can use iloc to select rows and columns by their integer position:

sales_data.iloc[:5, 2:] # first 5 rows, 3rd column onward

Understanding these subsetting methods is essential for filtering, slicing, and generally manipulating your data.

11. Adding new columns

Data exploration often involves creating new variables or features from existing ones. With pandas, adding a new column to a DataFrame is as easy as assigning it:

sales_data[‘revenue‘] = sales_data[‘price‘] * sales_data[‘sales‘]

This creates a new ‘revenue‘ column that is the product of the ‘price‘ and ‘sales‘ columns. Creating new columns like this allows you to engineer more informative features to use in your analysis.

12. set_index() and reset_index()

In some cases, you may want to change which column is used as the DataFrame‘s row index. You can do this with set_index():

sales_data.set_index(‘product_id‘, inplace=True)

This sets the ‘product_id‘ column as the new index. Conversely, you can use reset_index() to remove the index and turn it into a regular column:

sales_data.reset_index(inplace=True)

This removes ‘product_id‘ as the index and adds it as a column instead. Choosing an appropriate index for your DataFrame can make it easier to select, group, and merge data later on.

13. groupby()

GroupBy is one of the most powerful and frequently used operations in pandas. It allows you to split a DataFrame into groups based on one or more columns, then apply functions to each group. For example:

sales_data.groupby(‘category‘)[‘sales‘].mean()

Output:
category
clothing 454.186667
electronics 429.520000
home goods 427.366667
Name: sales, dtype: float64

This splits the DataFrame into groups based on the ‘category‘ column, then calculates the mean value of ‘sales‘ for each category. GroupBy makes it easy to aggregate and summarize data in all sorts of ways.

14. pivot_table()

Pivot tables are another powerful way to summarize and reshape data. The pivot_table() function allows you to group data by one or more columns and aggregate the values in another column. For example:

pd.pivot_table(sales_data, values=‘revenue‘, index=‘category‘, columns=‘region‘, aggfunc=np.sum)

This would create a pivot table showing the total revenue for each combination of product category and sales region. Pivot tables are a flexible way to explore relationships and generate useful summary tables from your data.

15. Plotting functions

Finally, no discussion of data exploration would be complete without mentioning visualization. Pandas integrates seamlessly with Matplotlib, allowing you to create plots directly from DataFrames using functions like plot(), hist(), and boxplot(). For example:

sales_data.groupby(‘category‘)[‘sales‘].plot(kind=‘bar‘)

This would generate a bar plot showing the total sales for each product category. Visualizations are a key part of exploring and understanding patterns, trends and relationships in your data.

Conclusion

Data exploration is a critical first step in any data analysis project. By taking the time to thoroughly examine and understand your dataset up front, you set yourself up for success in the later stages of analysis and modeling.

The pandas functions we‘ve covered here are some of the most useful tools for exploring and preprocessing data in Python. However, they‘re just a starting point – pandas has dozens of other functions and features for working with data.

The best way to get comfortable with data exploration in pandas is to practice on real datasets. Take the techniques you‘ve learned here and try applying them to your own data. Look for patterns, anomalies, and relationships. Visualize distributions and trends. Clean and reshape your data into a useful format for analysis.

With practice and experience, you‘ll develop an intuition for exploring data and uncovering valuable insights. You‘ll also discover more advanced pandas functions and techniques to add to your data science toolkit.

The field of data science is constantly evolving, but one thing remains constant – the importance of thoroughly exploring and understanding your data. Pandas is an indispensable tool for doing just that. So get out there and start exploring!