Column Non-Null Count Dtype

ByJordan Brown

When it comes to data analysis and manipulation in Python, the pandas library is an essential tool to master. Pandas provides powerful and flexible data structures like DataFrames and Series, along with a wide range of built-in functions to explore, clean, transform and analyze your data.

While there are hundreds of useful pandas functions, getting familiar with a core set will enable you to efficiently tackle most data analysis tasks. In this guide, we‘ll walk through 10 essential pandas functions you should know, with explanations and examples for each. We‘ll cover functions for data inspection, selection, missing data, summary statistics, sorting, and more.

Whether you‘re a pandas beginner or looking to fill in gaps in your knowledge, by the end of this post you‘ll be equipped with a powerful toolkit of pandas functions for streamlining your data analysis workflow. Let‘s dive in!

1. Inspecting Your Data

The first step in any data analysis project is getting to know your data. Pandas offers several convenient functions for quickly inspecting a DataFrame.

df.head() and df.tail()

To peek at the first or last rows of a DataFrame, use df.head() and df.tail(). By default these return the first or last 5 rows, but you can pass in a number to return that many rows:

df.head(3)
df.tail(10)

df.info()

To get a high-level summary of a DataFrame, use df.info(). This displays the number of rows, columns, column data types, and amount of non-null values:

df.info()



RangeIndex: 1000 entries, 0 to 999
Data columns (total 8 columns):





0   order_id          1000 non-null   int64

1   product           1000 non-null   object 
2   quantity          1000 non-null   int64

3   price             1000 non-null   float64
4   order_date        1000 non-null   object 
5   customer_id       1000 non-null   int64
6   country           1000 non-null   object 
7   is_repeat_order   1000 non-null   bool

dtypes: bool(1), float64(1), int64(3), object(4)
memory usage: 109.4+ KB

df.describe()

To see descriptive statistics for numerical columns like count, mean, standard deviation and quartile values, use df.describe():

df.describe()



   quantity         price    customer_id


count  1000.000000  1000.000000   1000.000000

mean      9.846000    23.764930   4945.337000

std       5.663451    14.847410   3453.991689

min       1.000000     1.000000      2.000000

25%       5.000000    11.690000   1498.750000

50%       9.000000    21.640000   4926.500000

75%      14.000000    33.657500   8515.000000

max      29.000000    99.000000   9999.000000




2. Selecting Subsets of Data


Pandas provides powerful methods to select subsets of your data based on labels, positions, or conditions.


df.loc[] and df.iloc[]


To select rows and columns by labels, use df.loc[]. To select by integer positions instead, use df.iloc[]:


df.loc[0:9, [‘product‘, ‘price‘]]
df.iloc[:5, 1:4]



df.query()


To select rows that match certain conditions, you can use df.query() and pass in a boolean expression as a string:


df.query(‘price > 50 and country == "Japan"‘)



3. Handling Missing Data


Real-world data often contains missing values that you‘ll need to detect and handle. Pandas makes this easy with functions like:


df.isnull()


To check for missing values, use df.isnull(). This returns a boolean mask the same shape as your DataFrame, with True for missing values:


df.isnull()



order_id  product  quantity   price  order_date  customer_id  country  is_repeat_order
0     False    False     False   False       False        False    False            False
1     False    False     False   False       False        False    False            False
2     False    False     False   False       False        False    False            False
3     False    False     False   False       False        False    False            False
4     False    False     False   False       False        False    False            False





df.fillna()


To fill in missing values, use df.fillna(). You can specify a single value to replace NaNs, or a dictionary mapping column names to values:


df.fillna(0)
df.fillna({‘price‘: df.price.mean(), ‘order_date‘: ‘2020-01-01‘}) 



4. Sorting Values


df.sort_values()


To sort a DataFrame by one or more columns, use df.sort_values(). Pass in the name of columns to sort by and set ascending=False to sort in descending order:


df.sort_values(‘price‘, ascending=False)
df.sort_values([‘country‘, ‘price‘])



5. Applying Functions to Data


A huge part of data analysis is applying custom functions and transformations to your data. Pandas has several key functions to enable this.


df.apply()


To apply a function along an axis of your DataFrame, use df.apply(). This is useful for transforming values, cleaning strings, or making complex calculations:


df[‘total_sales‘] = df.apply(lambda row: row[‘price‘] * row[‘quantity‘], axis=1) 
df.apply(np.mean, axis=0)  # mean of each column



df.applymap()


To apply a function element-wise to a whole DataFrame, use df.applymap():


df.applymap(lambda x: x.upper() if type(x) == str else x)



6. Merging and Grouping Data


Combining and aggregating data from multiple DataFrames or within groups is a common analysis task. 


df.merge()


To join two DataFrames on a shared column, use df.merge(). This provides SQL-style joins like inner, outer, left and right joins:


customer_df.merge(order_df, on=‘customer_id‘, how=‘inner‘)



df.groupby()


To group rows based on values in one or more columns, use df.groupby(). Then apply aggregate functions to get summary stats by group:


df.groupby(‘product‘).sum()
df.groupby([‘country‘, ‘product‘]).agg({‘quantity‘: ‘sum‘, ‘price‘: ‘mean‘})  



7. Reshaping Data


Pivoting and reshaping data is often necessary to get it into the right format for analysis or visualization.


df.pivot()


To reshape a DataFrame by specifying new index and column levels, use df.pivot():


df.pivot(index=‘country‘, columns=‘product‘, values=‘quantity‘) 



df.melt()


To unpivot or lengthen a DataFrame into a format with one row per observation, use df.melt():


df.melt(id_vars=[‘country‘], value_vars=[‘product‘, ‘quantity‘], var_name=‘category‘, value_name=‘value‘)



8. Visualizing Data


Pandas integrates with Matplotlib to easily create charts from DataFrames.


df.plot()


To make plots directly from a DataFrame, use the df.plot() method. Pass in parameters for plot kind, x and y values, color, and other formatting options:


df.groupby(‘product‘).sum()[‘quantity‘].plot(kind=‘bar‘)
df.plot(x=‘price‘, y=‘quantity‘, kind=‘scatter‘)



9. Handling Time Series Data


Pandas has robust tools for working with time series data like timestamps, date ranges, and frequencies.


pd.date_range()


To create a range of dates, use pd.date_range(). Specify a start and end date and optional frequency:


date_range = pd.date_range(start=‘2022-01-01‘, end=‘2022-12-31‘, freq=‘D‘)



df.resample()


To resample time series to a new frequency, use df.resample(). This is useful for aggregating or upsampling/downsampling data:


df.resample(‘M‘, on=‘order_date‘).sum()



10. IO and Data Sources


Pandas can read and write data from a variety of formats like CSV, Excel, SQL, JSON and more.


pd.read_csv()


To read a CSV into a DataFrame, use pd.read_csv(). Specify the file path and optional parameters for data types, delimiters, etc:


orders_df = pd.read_csv(‘orders.csv‘, parse_dates=[‘order_date‘])



df.to_sql()


To write a DataFrame to a SQL database, use df.to_sql(). Specify the database connection, table name, and if the table should be replaced or appended to:


df.to_sql(‘orders‘, engine, if_exists=‘replace‘, index=False)



Conclusion


In this guide, we covered 10 essential pandas functions for data analysis, including:


    

  1. Inspecting data with head(), tail(), info(), and describe() 
    2. 

  2. Selecting subsets with loc[], iloc[], and query()
    3. 

  3. Handling missing data with isnull() and fillna()
    4. 

  4. Sorting values with sort_values()
    5. 

  5. Applying functions with apply() and applymap()
    6. 

  6. Merging and grouping with merge() and groupby()
    7. 

  7. Reshaping data with pivot() and melt()
    8. 

  8. Visualizing data with plot()
    9. 

  9. Handling time series with date_range() and resample()  
    10. 

  10. Reading and writing data with read_csv() and to_sql()
    11. 



Of course, there are many more useful pandas functions to discover. But mastering these core tools will give you a strong foundation for efficient data analysis in Python.


To learn more, check out the official pandas documentation, practice with real-world datasets, and explore other great pandas resources like:


    

  • 10 Minutes to pandas 
    • 

  • Pandas Cookbook 
    • 

  • Practical Data Analysis with Python and Pandas
    • 

  • Python for Data Analysis
    • 



Happy data wrangling with pandas!


      
          
        How useful was this post?      
              
        Click on a star to rate it!      
        
      
                      
  •                       
    •                   
  •                       
    •                   
  •                       
    •                   
  •                       
    •                   
  •                       
    •               
    
    
        
      Average rating 0 / 5. Vote count: 0    
    
      No votes so far! Be the first to rate this post.    
    
  
          



You May Like to Read,

Similar Posts