Top Python Libraries to Automate Exploratory Data Analysis in 2021

As a data scientist, you know that exploratory data analysis (EDA) is a crucial step in any data science project. EDA is the process of exploring and analyzing a dataset to gain insights, spot anomalies, test hypotheses, and check assumptions before applying machine learning algorithms. A thorough EDA enables you to understand the structure and relationships in your data so you can effectively prepare it for modeling.

However, EDA can also be a time-consuming and tedious process, especially when working with large and complex datasets. In fact, data scientists often spend up to 60-80% of their time on data preparation tasks like data cleaning, formatting, transformation, and visualization. But what if there was a way to automate much of this work so you can quickly explore your data and move on to the fun part of building models?

Enter automated EDA tools. In recent years, a number of Python libraries have emerged that enable you to automate many common EDA tasks with just a few lines of code. These libraries can instantly generate detailed data profiles, interactive visualizations, and customized reports – saving you hours or even days of coding from scratch.

In this article, we‘ll take a look at the top Python libraries for automating exploratory data analysis in 2021. Whether you‘re a beginner looking for an easy way to explore your data or an experienced data scientist seeking to streamline your workflow, these tools can help you gain rapid insights from your data.

1. Pandas Profiling

Pandas Profiling is one of the most popular open-source libraries for automated EDA. With just a single line of code, Pandas Profiling can generate an interactive HTML report that provides a comprehensive overview of your dataset, including:

• Descriptive statistics like mean, median, range, and quantiles for each feature
• Information on missing values, zero values, and infinite values
• Histograms and bar charts showing the distribution of each feature
• Correlation matrices and scatterplots showing interactions between features
• Warnings about potential data quality issues like high cardinality, skewed distributions, and duplicate rows

Here‘s a quick example of how to use Pandas Profiling:

import pandas as pd
from pandas_profiling import ProfileReport

df = pd.read_csv(‘my_data.csv‘)
profile = ProfileReport(df, title=‘Pandas Profiling Report‘, html={‘style‘:{‘full_width‘:True}})
profile.to_file("my_report.html")

This code snippet will create an interactive HTML report called my_report.html that you can view in your web browser. The report includes a sidebar for quickly navigating different sections and visualizations.

One of the key advantages of Pandas Profiling is its ability to handle different data types, including numerical, categorical, boolean, and datetime features. It also provides smart default configuration settings while allowing customization of the analysis.

On the downside, Pandas Profiling can be resource-intensive and slow down when profiling very large datasets with millions of rows or hundreds of columns. Additionally, while it offers basic data cleansing functions like suppressing HTML warnings, it does not provide advanced capabilities for handling missing data, outliers, or inconsistent formats.

2. Sweetviz

Sweetviz is another excellent open-source library for generating beautiful, high-density visualizations to kickstart EDA. Its key differentiating features are:

• Ability to compare two datasets side-by-side (e.g. train vs test)
• Visualizations optimized for different data types and associations
• Highly customizable appearance and layout of the output report
• Support for understanding the impact of a target variable

Like Pandas Profiling, Sweetviz can be used with a single line of code:

import sweetviz as sv

report = sv.analyze(df)
report.show_html(‘sweetviz_report.html‘)

This will generate an interactive HTML report with the following sections:

• Dataset summary: Descriptive stats on the number of variables, observations, missing cells, memory size, etc.
• Variable summary: Table showing the type, unique values, missing values, and summary statistics for each feature
• Target analysis: Charts showing the distribution, feature associations, and importance of the target variable
• Categorical variables: Bar charts showing frequency counts and proportions
• Numerical variables: Histograms, KDE plots, box plots, and QQ plots
• Text analysis: Word clouds, top ngrams, and TFIDF

Sweetviz also makes it easy to compare multiple datasets, such as your training and test sets:

compare_report = sv.compare([train_df,‘Training Data‘], [test_df, ‘Test Data‘])

The resulting report will display visualizations for each dataset side-by-side along with the correlations between them.

Overall, Sweetviz provides an attractive, customizable interface for rapidly generating visualizations. However, it has fewer features compared to Pandas Profiling and is primarily focused on visualization rather than statistical analysis.

3. AutoViz

AutoViz is an intelligent tool that aims to generate the most relevant visualizations for your dataset based on the properties of the data itself. You can think of it as an AI assistant that helps determine the optimal charts for your EDA without requiring you to write the plotting code yourself.

The key features of AutoViz include:

• Automatic selection of visualizations based on feature datatypes and associations
• Best practice chart types including histograms, bar charts, scatterplots, line plots, heatmaps, and facet grids
• Handling of data preprocessing steps like scaling, normalization, and train/test splits
• Support for supervised and unsupervised analysis

Using AutoViz is incredibly simple:

from autoviz.AutoViz_Class import AutoViz_Class

AV = AutoViz_Class()
dfs = AV.AutoViz(‘my_data.csv‘)

AutoViz will return a dictionary of matplotlib plot objects which you can easily display in a Jupyter notebook or save to disk. It uses a smart algorithm to scan your data and determine the most relevant plot types – for example, histograms for univariate continuous features, bar charts for categorical variables, correlation heatmaps for bivariate analysis, facet plots for multivariate analysis and so on.

While AutoViz is very convenient, it has a couple of limitations. First, it currently only supports CSV files and does not allow you to pass in pandas dataframes. Second, the visualizations are static matplotlib images rather than interactive HTML reports. Finally, it does not offer as many statistical analysis and data cleansing functions as some other libraries.

4. D-Tale

D-Tale is a lightweight web application that provides an intuitive interface for exploring pandas dataframes. It combines the familiar functionality of pandas with interactive visualizations using a Flask backend and a React front-end.

Some of the key features of D-Tale are:

• Searchable/sortable/filterable dataframe viewer
• Univariate analysis with histograms, KDE plots, and box plots
• Bivariate analysis with scatter plots and hex bin plots
• Ability to build new columns using expressions
• Code export for reproducibility
• Integrations with popular data science libraries like scikit-learn and xgboost

To use D-Tale, you simply need to import it and pass a pandas dataframe to the show() function:

import dtale
import pandas as pd

df = pd.read_csv(‘my_data.csv‘) 
d = dtale.show(df)
d.open_browser()

This will automatically launch the D-Tale web app in your default browser. From there you can sort, filter, and visualize your dataframe to your heart‘s content. One nice feature is the ability to click on a column and view univariate analysis like value counts, descriptive statistics, string metrics (for object columns), and histograms.

D-Tale also makes it easy to build new columns from existing ones by writing pandas-like expressions. You can then export the updated dataframe or a sample of it to continue your analysis elsewhere. Finally, D-Tale even offers built-in machine learning with one-click access to classification/regression models, clustering, and feature analysis.

The main downside of D-Tale is that it requires a bit more setup compared to libraries that just generate a static report. However, the interactivity it enables can greatly speed up your EDA workflow once you get it up and running.

5. DataPrep

DataPrep is an open-source library from SFU Data Science Research Group that combines data loading, cleaning, normalization, transformation, and visualization. It provides a high-level, unified interface for quickly exploring and preparing data for analysis and modeling.

Some highlights of DataPrep are:

• Automatic data type detection and parsing
• Built-in functions for handling missing values, outliers, and inconsistent data
• Intelligent visual recommendations based on data characteristics
• Suggestive data transformations like encoding, scaling, and normalization
• Text cleaning and feature extraction
• Integration with Pandas, Dask, and third-party visualization libraries

DataPrep emphasizes a functional, pipeline-oriented approach that allows you to rapidly iterate through the stages of data prep and EDA. Here‘s an example:

from dataprep.eda import *
from dataprep.datasets import load_dataset

df = load_dataset(‘titanic‘)
plot(df, plot_type=‘histogram‘)

This code will automatically generate a grid of histograms showing the distribution of each feature in the famous Titanic dataset. You can easily switch the plot_type to other univariate charts like KDE or box plots.

DataPrep also provides smart data cleaning functions that can be chained together, for example:

df = df.clean() \
       .impute_missing() \ 
       .normalize_numeric() \
       .encode_categorical()  

This will automatically remove duplicate rows, impute missing values, standardize numeric columns and one-hot/label encode categorical columns. DataPrep uses intelligent heuristics to determine the appropriate cleaning and transformation steps.

While DataPrep is a powerful tool for data preparation, it is a relatively new library and does not yet have the extensive ecosystem and community support of more established projects. Additionally, some of the automated data cleaning steps may not be suitable for every use case.

6. Lux

Lux is a novel Python library that combines automated EDA with interactive visualization recommendation. It allows you to quickly discover visual insights from your pandas dataframes without having to manually specify the visualization type or write any matplotlib code.

The key idea behind Lux is the concept of "intent". By parsing the pandas operations you perform on a dataframe, Lux infers your analysis intent and recommends appropriate visualizations. For example, if you groupby a column and compute the mean, Lux will display a bar chart of the means by group. If you sort_values, Lux will show you the top and bottom records. If you corr, Lux will suggest a heatmap of the correlations, and so on.

Using Lux is as simple as importing it and turning on the default display:

import lux
import pandas as pd

df = pd.read_csv(‘my_data.csv‘) 
df.intent = [""]
df

This will display the Lux widget below your dataframe, showing the top recommended visualizations based on the current state of the dataframe. You can click on the different options to switch between chart types, variables, and aggregations. Lux even provides a powerful natural language interface, allowing you to type queries like "show correlation between mpg and weight" or "plot histogram of Sepal.Length".

Lux is a great tool for interactively exploring your data and uncovering hidden insights. It can save you a lot of time that would otherwise be spent writing redundant visualization code. However, it is still an experimental project and may not be suitable for production use cases. It also requires a Jupyter notebook environment and does not generate standalone HTML reports like some of the other libraries.

Conclusion

Exploratory data analysis is an essential part of any data science project, but it doesn‘t have to be a chore. By leveraging the power of automated EDA libraries, you can quickly gain a comprehensive understanding of your dataset and identify the most promising areas for further analysis.

Whether you prefer generating static HTML reports or interactive web-based tools, there is a Python library out there to streamline your EDA workflow. Pandas Profiling and Sweetviz are great for rapidly profiling your data, while AutoViz provides automatic visualization recommendations. D-Tale and Lux offer more interactive, notebook-style interfaces for iterative exploration. Finally, DataPrep combines data loading, cleaning, and visualization into a unified pipeline.

No matter which tool you choose, automated EDA can help you spend less time writing boilerplate code and more time extracting valuable insights from your data. So why not give them a try on your next data science project?