Data Visualization: An AI/ML Expert‘s Guide to Exploring Data with Scatter Plots, Histograms, and More

Data visualization is a critical skill for any data scientist, analyst, or AI/ML practitioner. By creating visual representations of data, we can quickly identify patterns, trends, outliers, and relationships that may not be apparent from looking at numbers alone. Effective visualization helps guide feature engineering, model selection, and hyperparameter tuning. It allows us to evaluate model performance, explain results to stakeholders, and catch problems like bias and overfitting.

Research has repeatedly shown that human brains process visual information much better than text or numbers. One study found that people retain only 10-20% of written or spoken information but 65% of visual information[^1]. Another showed that presentations using visual aids were 43% more persuasive than those without[^2]. In short, the human brain is wired for visual learning – which is why data visualization is so important!

Histograms and Scatter Plots: The Bread and Butter of Data Viz

While there are dozens of chart types out there, two of the most essential for data exploration are histograms and scatter plots.

Histograms: Understanding Distributions

A histogram shows the distribution of a single numerical variable. It splits the data into "bins" (intervals) and shows the count or frequency of data points in each bin. Here‘s an example created with Seaborn:

import seaborn as sns
titanic = sns.load_dataset(‘titanic‘)
sns.histplot(data=titanic, x=‘age‘, bins=30)

From this, we can see that the age distribution on the Titanic was slightly right-skewed, with a peak around 20-30 years old. There are potential outliers on the high end.

Histograms help answer questions like:

• What is the shape of the distribution? Symmetric? Skewed?
• Are there multiple modes (peaks)?
• What is the typical value? The median? The range?
• Any suspected outliers on the low or high end?

Examining the distribution of each variable is an important first step in exploratory data analysis. Distributions that are highly skewed or contain outliers may need to be transformed (e.g. log scale) or have outliers removed before feeding into machine learning models.

Scatter Plots: Visualizing Relationships

A scatter plot visualizes the relationship between two continuous variables. Each (x,y) point represents one observation. Here‘s an example looking at the relationship between horsepower and mpg in the classic "Auto MPG" dataset:

import matplotlib.pyplot as plt
import seaborn as sns

auto = sns.load_dataset(‘mpg‘)
sns.scatterplot(data=auto, x=‘horsepower‘, y=‘mpg‘, size=‘weight‘, hue=‘origin‘, style=‘origin‘)
plt.title(‘Horsepower vs. MPG‘)

The downward trend indicates that cars with higher horsepower tend to have lower fuel efficiency (mpg). The points are colored and styled by country of origin, revealing that American cars tend to have higher horsepower but lower efficiency compared to Japanese and European vehicles. Point size represents vehicle weight.

Scatter plots help uncover relationships and generate hypotheses:

• Is there a correlation between the variables? Positive or negative?
• How strong is the relationship? Is it linear?
• Are there subgroups or clusters in the data?
• Any notable outliers?

Scatter plots are also useful for visualizing model performance, such as actual vs. predicted values, residuals vs. fitted values, or accuracy vs. a hyperparameter.

The Curse of Dimensionality and Manifold Learning

Real-world datasets often have dozens or even hundreds of variables. This is a challenge for visualization, as plots are inherently 2D or at most 3D. High-dimensional data often suffers from the "curse of dimensionality," where the volume of the space increases exponentially with the number of dimensions[^3]. This makes it harder to visualize and comprehend the data.

One solution is dimensionality reduction – projecting the data to a lower-dimensional space while preserving key structure. A popular technique for this is t-SNE (t-Distributed Stochastic Neighbor Embedding)[^4]. t-SNE maps high-dimensional data to 2 or 3 dimensions such that similar points are near each other and dissimilar points are far apart.

Here‘s an example using t-SNE to visualize handwritten digits:

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE

digits = load_digits()
tsne = TSNE(n_components=2, random_state=1)
digits_tsne = tsne.fit_transform(digits.data)

plt.figure(figsize=(8, 8))
plt.scatter(digits_tsne[:, 0], digits_tsne[:, 1], c=digits.target, cmap=‘jet‘)
plt.colorbar(label=‘digit‘)
plt.title(‘t-SNE of Handwritten Digits‘)

The t-SNE plot reveals that the digits form clusters, with some overlap between visually similar digits like 1 and 7. This type of plot is useful for seeing structure in high-dimensional datasets and checking class separability before training a classifier.

Visualizing for Fairness and Bias Detection

As AI and machine learning systems become more prevalent in high-stakes domains like healthcare, criminal justice, and lending, ensuring that models are fair and unbiased is critical. Visualization can help uncover disparities and biases in data and model outputs.

For example, consider a model that predicts whether a loan applicant will default. We can visualize the distribution of model scores for different demographic groups using violin plots:

import numpy as np
import pandas as pd

scores_male = np.random.normal(0.7, 0.2, 1000)
scores_female = np.random.normal(0.4, 0.3, 1000)

scores_df = pd.DataFrame({
    ‘score‘: np.concatenate([scores_male, scores_female]),
    ‘gender‘: [‘male‘] * 1000 + [‘female‘] * 1000
})

sns.violinplot(data=scores_df, x=‘gender‘, y=‘score‘)
plt.title(‘Credit Default Model Scores by Gender‘)

The plot shows that the model score distribution is shifted higher for males than for females. This suggests the model may be biased, assigning higher default risk to women. Disparities like this need to be investigated and mitigated before deploying the model.

Another fairness visualization is the confusion matrix, which shows a model‘s true vs. predicted classifications broken down by sensitive attribute:

from sklearn.metrics import confusion_matrix
import seaborn as sns

males_cf = confusion_matrix(male_true, male_pred)
females_cf = confusion_matrix(female_true, female_pred)

plt.figure(figsize=(12, 5))
plt.suptitle("Confusion Matrices by Gender")

plt.subplot(121)
sns.heatmap(males_cf, annot=True, fmt=‘g‘, cmap=‘Blues‘, cbar=False)
plt.xlabel(‘Predicted Class‘)
plt.ylabel(‘True Class‘)
plt.title(‘Male Applicants‘)

plt.subplot(122) 
sns.heatmap(females_cf, annot=True, fmt=‘g‘, cmap=‘Blues‘, cbar=False)
plt.xlabel(‘Predicted Class‘)
plt.ylabel(‘True Class‘) 
plt.title(‘Female Applicants‘)

plt.tight_layout()
plt.show()

The confusion matrices reveal that the model has higher accuracy for male applicants (93%) than female applicants (85%), another red flag for bias.

Advanced Visualization Techniques

Beyond the basics of histograms and scatter plots, there are many advanced visualization techniques that can yield deeper insights into data and models:

• Parallel Coordinates: Useful for visualizing multivariate data and spotting patterns and outliers.
• Radar Charts: Good for comparing multiple quantitative variables across a few items or groups.
• Network Graphs: Visualize relationships or flows between items; useful for social network or customer segmentation analysis.
• Alluvial Diagrams: Show changes in flow or composition over time or across categories.
• Geospatial Maps: Plot geographic data like GPS coordinates, census tracts, or store locations on interactive maps.

Tools like D3.js, Plotly, and Bokeh support many of these advanced chart types and allow for interactive features like zooming, filtering, and animating.

Best Practices for Effective Visualization

To ensure your data visualizations are accurate, engaging, and impactful, follow these best practices:

1. Choose the right visual for your data and question. Don‘t use a pie chart to compare 20 categories or a line chart for non-time series data.

2. Maximize the data-ink ratio. Remove unnecessary borders, gridlines, labels, and legends. Every bit of ink should serve a purpose.

3. Use color sparingly and intentionally. Limit your palette to a few contrasting colors to highlight key categories or values.

4. Label and annotate with care. Provide informative titles, axis labels, and captions. Highlight key insights with annotations.

5. Design for your audience. A visualization for executives should look different than one for data scientists. Know your audience and customize the design and level of detail accordingly.

6. Make it interactive. Allow users to zoom in and out, filter data, highlight points, and see details on demand. Interactivity boosts engagement and exploration.

7. Tell a story. A compelling visualization doesn‘t just show data – it uses data to tell a story. What is the key message or narrative that ties the data together?

With thoughtful design and execution, your data visualizations can be powerful tools for driving insight and action.

Conclusion

Data visualization is a superpower for AI and machine learning practitioners. From exploratory data analysis to model evaluation to stakeholder communication, clear and compelling visuals are essential at every stage of the workflow.

Histograms and scatter plots are fundamental chart types every data scientist should have in their toolkit. But don‘t stop there – learn advanced techniques like t-SNE, parallel coordinates, and geospatial mapping to take your visualizations to the next level.

Remember, with great power comes great responsibility. Use your visualization skills for good – to detect bias, ensure fairness, and tell stories that matter. The human visual system is a wonder of evolution – let‘s leverage it to build smarter, safer, and more impactful AI systems.

Now go forth and visualize!

[^1]: Trafton, A. (2014). In the blink of an eye. MIT News. http://news.mit.edu/2014/in-the-blink-of-an-eye-0116

[^2]: Vogel, D. R., Dickson, G. W., & Lehman, J. A. (1986). Persuasion and the role of visual presentation support: The UM/3M study. Working Paper Series (MISRC-WP-86-11). Management Information Systems Research Center, School of Management, University of Minnesota.

[^3]: Bellman, R. E. (1961). Adaptive control processes – A guided tour. Princeton, NJ: Princeton University Press.

[^4]: van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86), 2579-2605.