# A Deep Dive into 2D Density Plots for Exploratory Data Analysis in Python: An AI/ML Perspective

- Canonical: https://33rdsquare.com/fundamentals-of-exploratory-data-analysis/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Exploratory Data Analysis, or EDA, is a critical first step in any data science project, but it‘s especially crucial in the world of artificial intelligence and machine learning. Before even thinking about training a sophisticated deep learning model or building a predictive analytics pipeline, data scientists need to develop a comprehensive understanding of the data they‘re working with.

EDA is the process of exploring and analyzing datasets to uncover insights, patterns, and anomalies that can inform feature engineering, model selection, and other key aspects of the AI/ML workflow. Neglecting this step can lead to a host of problems downstream, from training on irrelevant features to deploying models that fail in the real world.

Just how important is EDA in practice? A [2020 Anaconda survey](https://www.anaconda.com/state-of-data-science-2020) of over 2,300 data scientists found that they spend nearly 40% of their time on data preparation and data cleansing tasks. [Another survey](https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/) by CrowdFlower revealed that 60% of data scientists consider data preparation to be the least enjoyable part of their job. By investing time upfront in EDA, data scientists can not only make their analyses more efficient and effective, but also potentially uncover game-changing insights.

So what exactly does EDA entail? At a high level, it involves both statistical and visual methods to describe and summarize datasets. Some key EDA techniques include:

| Technique | Description | When to Use |
| --- | --- | --- |
| Univariate visualization | Plotting individual variables with histograms, box plots, etc. | Understanding distributions, detecting outliers |
| Bivariate visualization | Comparing two variables with scatter plots, line plots, etc. | Identifying relationships and correlations |
| Multivariate visualization | Analyzing 3+ variables with pair plots, parallel coordinates, etc. | Exploring high-dimensional data, detecting patterns |
| Descriptive statistics | Calculating means, medians, standard deviations, etc. | Summarizing and comparing variables |
| Correlation analysis | Measuring statistical relationships between variables | Identifying predictors, informing feature selection |
| Dimensionality reduction | Reducing high-dimensional data to 2-3 dimensions | Visualizing complex datasets, identifying latent factors |

While all of these techniques have their place in the EDA process, one of the most powerful tools for analyzing the relationship between two continuous variables is the 2D density plot. These plots show the estimated joint probability density function of two variables, revealing not just the location of data points but also their relative concentration. Density plots offer several advantages over simple scatter plots or histograms:

- They avoid the visual clutter of plotting every single data point
- They aren‘t sensitive to the choice of bin width or anchor points
- They provide a smooth, continuous representation of the full distribution
- They can be easily faceted or color-coded to visualize different subgroups

Creating 2D density plots in Python is straightforward thanks to visualization libraries like Matplotlib and Seaborn. Here‘s a simple example using Seaborn to create a density plot of the famous Iris dataset:

```
import seaborn as sns
import matplotlib.pyplot as plt

iris = sns.load_dataset(‘iris‘)

sns.kdeplot(data=iris, x=‘sepal_length‘, y=‘sepal_width‘,
            hue=‘species‘, fill=True, common_norm=False,
            palette=‘crest‘, alpha=.5, linewidth=0)

plt.show()
```

This code loads the Iris dataset and creates a density plot with sepal length on the x-axis, sepal width on the y-axis, and species encoded by color. The `fill` parameter fills in the contours, while `common_norm=False` scales each species density separately. The result is a visually appealing plot that clearly shows the different distributions of the three Iris species:

![Iris Density Plot](https://33rdsquare.com/iris-density-plot.png)

While this example covers the basics, there are many ways to customize density plots for deeper insights. One powerful technique is to use contour plots to visualize specific density thresholds. Here‘s an example using the same Iris data:

```
sns.kdeplot(data=iris, x=‘sepal_length‘, y=‘sepal_width‘,
            hue=‘species‘, thresh=.2, levels=4,
            fill=False, linewidths=2)

plt.show()
```

Setting `thresh=.2` and `levels=4` draws 4 contour lines at 20% density intervals for each species. The result highlights the peak density regions and reveals some interesting patterns, like the elongated distribution of Iris-setosa compared to the more circular distributions of the other species.

![Iris Contour Plot](https://33rdsquare.com/iris-contour-plot.png)

Density plots can also be extremely useful for anomaly and outlier detection, a common task in AI/ML projects. By plotting the density of "normal" data and overlaying new data points, it‘s easy to spot values that fall in low-density regions. Here‘s an example of using density plots for anomaly detection on the [MNIST handwritten digits dataset](http://yann.lecun.com/exdb/mnist/):

```
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA

digits = load_digits()
X_train, X_test = train_test_split(digits.data, random_state=1)

pca = PCA(n_components=2).fit(X_train)
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)

sns.kdeplot(x=X_train_pca[:,0], y=X_train_pca[:,1],
            shade=True, shade_lowest=False,
            palette=‘Purples_r‘, alpha=0.5)

sns.scatterplot(x=X_test_pca[:,0], y=X_test_pca[:,1], s=50)

plt.show()
```

This code loads the MNIST digits dataset, splits it into train and test sets, and uses PCA to reduce the 64-dimensional pixel data to 2 dimensions. It then plots the density of the training data and overlays the test points as a scatter plot. The result shows several test digits that fall in very low-density regions of the training distribution, indicating that they are likely anomalies or outliers:

![MNIST Anomaly Detection](https://33rdsquare.com/mnist-anomaly-detection.png)

In a real ML project, these anomalous examples could represent data points that are mislabeled, corrupted, or simply very different from the training data. Detecting and investigating these anomalies is crucial for ensuring the quality and reliability of the final trained model.

Beyond just visualizing distributions, density plots can also play a key role in feature selection and engineering. By comparing the density of different variables conditioned on the target variable, data scientists can identify the most discriminative features and potentially uncover important interaction effects.

For example, consider the classic [Adult Census Income dataset](https://archive.ics.uci.edu/ml/datasets/adult), where the goal is to predict whether an individual earns over $50,000 per year based on demographic and employment factors. Here‘s how density plots can reveal the predictive power of different features:

```
import pandas as pd

df = pd.read_csv(‘adult.csv‘)

sns.kdeplot(data=df, x=‘age‘, hue=‘income‘,
            fill=True, common_norm=False, alpha=.5);
plt.show()

sns.kdeplot(data=df, x=‘education-num‘, hue=‘income‘,
            fill=True, common_norm=False, alpha=.5);
plt.show()
```

The density plots show that age has very similar distributions for high and low income individuals, while education level is more differentiated. This suggests that education may be a more informative predictor than age for this task.

![Adult Age Density Plot](https://33rdsquare.com/age-density-plot.png)
 ![Adult Education Density Plot](https://33rdsquare.com/education-density-plot.png)

Of course, density plots are just one piece of the puzzle when it comes to feature selection. Data scientists also need to consider factors like feature redundancy, interpretability, and computational efficiency. But visualizing the conditional densities is a great way to quickly identify promising candidates and generate hypotheses for further analysis.

So far we‘ve focused on 2D density plots, but what about visualizing higher-dimensional data? While it‘s impossible to directly plot densities in more than three dimensions, there are some clever techniques for visualizing multivariate distributions. One approach is to use small multiples of 1D or 2D densities conditioned on different variables. Another is to use dimensionality reduction methods like PCA, t-SNE, or UMAP to project the data to a lower-dimensional space while preserving the salient density structure.

Here‘s an example of using small multiples to visualize the 4D Iris data by species and sepal area:

```
iris[‘sepal_area‘] = iris[‘sepal_length‘] * iris[‘sepal_width‘]

g = sns.PairGrid(iris, hue=‘species‘, diag_sharey=False)
g.map_lower(sns.kdeplot, fill=True)
g.map_diag(sns.kdeplot, fill=True)
g.add_legend()

plt.show()
```

![Iris Small Multiples](https://33rdsquare.com/iris-small-multiples.png)

The grid of density plots allows us to see both the univariate density of each variable by species, as well as the bivariate relationships between sepal length, width, and area. We can see, for example, that Iris-setosa has smaller sepals overall and that sepal length and width are correlated for all three species.

For even higher-dimensional data, dimensionality reduction is often the only viable approach. Here‘s an example of using UMAP to visualize the density of the 784-dimensional MNIST dataset in 2D:

```
from umap import UMAP

digits = load_digits()

umap = UMAP(n_components=2).fit_transform(digits.data)

sns.kdeplot(x=umap[:,0], y=umap[:,1], fill=True, thresh=0.05,
            levels=10, cmap=‘magma_r‘)

plt.show()
```

![MNIST UMAP Density](https://33rdsquare.com/mnist-umap-density.png)

The UMAP projection preserves both the local and global structure of the digit images, revealing distinct density peaks for each digit class. Outliers and lower-density regions are also visible, providing a useful visual summary of the entire dataset.

As these examples illustrate, density plots are an incredibly versatile tool for AI/ML practitioners looking to understand and explore their data. Whether using them for anomaly detection, feature selection, or high-dimensional visualization, density plots provide a unique window into the structure and relationships hidden in complex datasets.

Of course, density plots are just one arrow in the EDA quiver. Responsible data scientists know that generating true insights requires synthesizing multiple perspectives using a variety of statistical and visual techniques. They also know that EDA is not a one-time exercise, but rather an iterative process of questioning, exploring, and refining hypotheses.

As AI and ML continue to advance, the role of EDA will only become more critical. With the rise of deep learning and advanced analytics, it‘s easier than ever to simply throw data at a black box model and hope for the best. But truly reliable and impactful solutions require a deep understanding of the domain, the data, and the underlying assumptions. Exploratory analysis is the key to developing that understanding.

Looking forward, I believe that EDA will increasingly be seen not as a burdensome chore, but as an opportunity for creativity and discovery. New interactive visualization tools and automated EDA platforms will help streamline the process, while still leaving room for human insight and intuition. At the same time, the explosive growth of AI/ML will create new challenges and opportunities for EDA, from analyzing massive unstructured datasets to explaining and debugging complex models.

As data scientists, it‘s up to us to embrace these challenges and lead the way in extracting knowledge from data. By mastering the art and science of exploratory analysis, we can not only build better AI/ML solutions, but also uncover the hidden stories and insights that have the power to transform organizations and change the world.

## Additional Resources

To learn more about EDA and density plots in the context of AI/ML, check out these resources:

- [Why EDA is Crucial in Data Science and Machine Learning Projects](https://towardsdatascience.com/why-eda-is-crucial-in-data-science-and-machine-learning-projects-6db0a4f8e590) (Medium)
- [The Ultimate Guide to Data Exploration](https://www.maartengrootendorst.com/blog/eda/) (Maarten Grootendorst)
- [A Gentle Introduction to Probability Density Estimation](https://machinelearningmastery.com/probability-density-estimation/) (Machine Learning Mastery)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://umap-learn.readthedocs.io/en/latest/) (Documentation)
- [Outlier detection with Kernel Density Functions](https://scikit-learn.org/stable/auto_examples/neighbors/plot_kde_outlier_detection.html) (Scikit-Learn)

---

Source: [A Deep Dive into 2D Density Plots for Exploratory Data Analysis in Python: An AI/ML Perspective](https://33rdsquare.com/fundamentals-of-exploratory-data-analysis/)