# The Normal Distribution: A Data Scientist‘s Best Friend

- Canonical: https://33rdsquare.com/statistics-data-science-normal-distribution/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

The normal distribution is the Bruce Wayne of probability distributions – a billionaire playboy on the outside but a superhero vigilante on the inside. By day it‘s an unassuming bell curve, but by night it‘s the secret power behind much of statistics and machine learning. If you‘re an aspiring data scientist or AI master, you‘ll need to be best buds with the normal distribution. Let‘s unmask this mysterious statistical superhero!

## The Man Behind the Normal Distribution

First off, some history – who is the Tony Stark behind this iconic Bell curve of justice? The normal distribution was actually discovered by two brilliant mathematicians in the early 18th century:

- Abraham de Moivre, an English mathematician who first discovered the normal distribution in 1733 as an approximation to the binomial distribution for large n [1]. He was a rock star of probability theory and is famously quoted as saying "The normal distribution is a probability distribution."
- Pierre-Simon Laplace, a French mathematician who used the normal distribution in 1774 to analyze errors in astronomical observations [2]. Not only was he a math genius but also apparently a part-time astronomer – talk about a renaissance man!

Today, the normal distribution is also known as the Gaussian distribution after the famous German mathematician Carl Friedrich Gauss (he‘s like the Beyoncé of math – so famous he only needs one name). Gauss used the normal distribution in 1809 to analyze astronomical data and proved some of its important properties [2].

## The Heroic Properties and Equations of the Normal Distribution

What makes the normal distribution so super? For starters, check out its probability density function (PDF):

$$ f(x) = \frac{1}{\sigma \sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$

where:

- $\mu$ is the mean or expectation
- $\sigma$ is the standard deviation
- $\sigma^2$ is the variance

A few more of its heroic properties:

1. Symmetry: The normal curve is symmetric about the mean, which equals the median and mode.
2. Unimodality: There is a single peak at the mean (no alter egos here!).
3. Asymptotic: The curve approaches but never touches the horizontal axis on either side.
4. Empirical Rule: Approximately 68% of the data falls within 1 standard deviation of the mean, 95% within 2 standard deviations, and 99.7% within 3 standard deviations.
5. Central Limit Theorem: The sum of a large number of independent variables will be approximately normally distributed, regardless of their individual distributions. More on this later!

Here are some quick facts about the normal distribution [2]:

| Property | Value |
| --- | --- |
| Support | $(-\infty, \infty)$ |
| Mean | $\mu$ |
| Median | $\mu$ |
| Mode | $\mu$ |
| Variance | $\sigma^2$ |
| Skewness | 0 |
| Kurtosis | 3 |
| MGF | $\exp(\mu t + \frac{1}{2}\sigma^2 t^2)$ |

## The Secret Identity of the Normal Distribution in AI/ML

So why do AI and machine learning experts love the normal distribution more than Superman loves saving damsels in distress? Here are some of its top secret identities in AI/ML:

1. Maximum Likelihood Estimation (MLE) – Many AI/ML models use MLE to estimate the model parameters that maximize the probability of the observed data. For a large class of models, the MLE solution relies on normally distributed errors or residuals. For example, in linear regression, we assume the errors are i.i.d. normal:

$$ y_i = \beta_0 + \beta_1 x_i + \epsilon_i $$

where $\epsilon_i \sim N(0,\sigma^2)$. This normality assumption allows us to find the least squares estimates of $\beta_0$ and $\beta_1$.

1. Bayesian Machine Learning – In Bayesian ML, we use probability distributions to represent uncertainty in model parameters and predictions. The normal distribution is commonly used as a prior or posterior distribution due to its convenient conjugate properties. For instance, a normal prior with a normal likelihood yields a normal posterior.
2. Neural Networks – In deep learning, we often initialize the weights of a neural network with small random values drawn from a normal distribution. This helps break symmetry and allows the network to learn different features. The choice of the normal distribution here is somewhat arbitrary but mathematically convenient. The activations and gradients of a neural network also tend to follow normal distributions by the central limit theorem [3].
3. Anomaly Detection – Many anomaly detection algorithms assume the normal data comes from a multivariate Gaussian distribution and identify outliers as points with low probability under this distribution. For example, the Mahalanobis distance measures how many standard deviations away a point is from the mean of a distribution.
4. Natural Language Processing – In NLP, words are often represented as high-dimensional vectors called word embeddings. Popular word embedding models like word2vec and GloVe generate these vectors by assuming words follow a multivariate normal distribution. For example, GloVe optimizes the objective [4]:

$$ J = \sum_{i,j=1}^V f(X_{ij}) (\log(X_{ij}) – \log(\hat{X}_{ij}))^2 $$

where $X_{ij}$ is the co-occurrence count of words $i$ and $j$, $\hat{X}_{ij} = \exp(w_i^T \tilde{w}_j + b_i + \tilde{b}_j)$ is the model estimate, and $f$ is a weighting function. The word vectors $w_i$ and $\tilde{w}_j$ are assumed to be normally distributed.

1. Generative Adversarial Networks (GANs) – GANs are a popular framework for generating realistic images, videos, and other data. The generator network in a GAN typically takes random noise drawn from a normal distribution as input and transforms it into a realistic sample. The normal distribution provides a smooth, continuous latent space that the generator can map to the data space.

Absolutely incredible! The normal distribution is the hero that AI/ML needs, even if it‘s not the one it deserves.

## The Dark Side of Normality Assumptions

Now, every superhero has a weakness (except Chuck Norris), and the normal distribution is no exception. Here are some limitations and pitfalls of assuming normality in statistics and ML:

1. Skewness and Heavy Tails – Many real-world datasets are skewed or have heavy tails that deviate significantly from normality. In these cases, using normal-based models can lead to biased parameter estimates, inflated type I error rates, and poor predictive performance [5].
2. Mixtures and Multimodality – Some datasets come from a mixture of multiple distributions or have multiple modes. Trying to fit a single normal distribution to such data is like trying to fit a square peg in a round hole – it just doesn‘t work!
3. Outliers and Contamination – Outliers can severely distort the mean and variance estimates of a normal distribution. Robust statistics are needed to handle outliers and contaminated data.
4. Small Sample Sizes – The central limit theorem only holds for large sample sizes. For small samples, the sampling distribution of the mean may not be exactly normal, and normal-based inference can be inaccurate.
5. Heteroscedasticity – Many statistical and ML models assume homoscedasticity, i.e., constant variance of errors. If the variance is not constant (heteroscedasticity), the model assumptions are violated and the results may be invalid.

So how do we diagnose and address these issues? Here are some tips:

- Plot the data! Histograms, Q-Q plots, and box plots can reveal skewness, heavy tails, outliers, and multimodality.
- Use robust summary statistics like median, IQR, and trimmed mean instead of mean and standard deviation.
- Transform the data to be more normal, e.g., by taking the log, square root, or Box-Cox transformation.
- Use multilevel, mixture, or nonparametric models to handle non-normality and heterogeneity.
- Use cross-validation or bootstrap to assess model performance and variability.

## Case Study: Exploring Normality in the Iris Dataset

Let‘s put on our data science cape and apply our normality superpowers to the famous Iris dataset. This dataset contains measurements of sepal length, sepal width, petal length, and petal width for three species of iris flowers. Our superhero mission is to explore whether these measurements are normally distributed! Here are the steps:

Step 1 – Load the libraries and data:

```
from sklearn.datasets import load_iris
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

iris = load_iris()
X = iris.data
y = iris.target
cols = iris.feature_names
df = pd.DataFrame(X, columns=cols)
df[‘species‘] = pd.Series(y).map(dict(zip(range(3),iris.target_names)))
```

Step 2 – Check summary statistics:

```
df.describe()
```

|  | sepal length (cm) | sepal width (cm) | petal length (cm) | petal width (cm) |
| --- | --- | --- | --- | --- |
| count | 150.000000 | 150.000000 | 150.000000 | 150.000000 |
| mean | 5.843333 | 3.054000 | 3.758667 | 1.198667 |
| std | 0.828066 | 0.433594 | 1.764420 | 0.763161 |
| min | 4.300000 | 2.000000 | 1.000000 | 0.100000 |
| 25% | 5.100000 | 2.800000 | 1.600000 | 0.300000 |
| 50% | 5.800000 | 3.000000 | 4.350000 | 1.300000 |
| 75% | 6.400000 | 3.300000 | 5.100000 | 1.800000 |
| max | 7.900000 | 4.400000 | 6.900000 | 2.500000 |

The mean and median are quite close for sepal length and width, but differ for petal length and width, which may indicate skewness.

Step 3 – Visualize the distributions:

```
fig, axs = plt.subplots(2, 2, figsize=(10,10))
axs = axs.flatten()

for i, col in enumerate(cols):
    ax = axs[i]
    ax.hist(df[col], bins=20, density=True, alpha=0.5)
    ax.set_title(col)

    mn, mx = df[col].min(), df[col].max()
    x = np.linspace(mn, mx, 100)
    p = stats.norm.pdf(x, loc=df[col].mean(), scale=df[col].std())

    ax.plot(x, p, ‘r--‘, linewidth=2)
    ax.set_xlabel(col)
    ax.set_ylabel(‘Density‘)

plt.tight_layout()
plt.show()
```

![Iris Histograms and PDFs](https://i.imgur.com/nXIcTBQ.png)

Sepal length and width look approximately normal, but petal length and width are right-skewed. The normal curve (red dashed line) does not fit the histograms well for petal measurements.

Step 4 – Perform Shapiro-Wilk test:

```
for col in cols:
    print(f‘{col}: {stats.shapiro(df[col])}‘)
```

```
sepal length (cm): (0.9775081872940063, 0.01547532342374325)
sepal width (cm): (0.9775206446647644, 0.01560454070568085)
petal length (cm): (0.8761621117591858, 5.694532123190177e-08)
petal width (cm): (0.9016568064689636, 3.3253581372239935e-07)
```

The Shapiro-Wilk test rejects the null hypothesis of normality for petal length and width (p < 0.05) but not for sepal length and width (p > 0.05).

In summary, our data science sleuthing has uncovered that the sepal measurements are normally distributed, but the petal measurements are not. We may need to transform the petal variables or use nonparametric methods if we want to analyze this dataset using models that assume normality.

## Conclusion

Phew, what a thrilling adventure through the wonderful world of normal distributions! We‘ve unmasked the secret identities of the normal distribution in statistics and machine learning, discovered its superhero origin story, marveled at its mighty equations, learned its heroic properties, and battled its villainous pitfalls.

But with great statistical power comes great responsibility! As data scientists and AI avengers, it‘s our duty to use the normal distribution wisely and judiciously. Always check assumptions, visualize the data, and consider alternatives when the normal distribution doesn‘t fit. And never underestimate the power of the central limit theorem and Z-scores to save the day!

So the next time you encounter a dataset claiming "I‘m normally distributed!", you‘ll know exactly how to respond: "Nice try, but I‘m a trained data science ninja with a black belt in distribution-fu. I‘ve got my eye on you, mister."

## References

[1] Hazewinkel, M. (Ed.). (2001). Normal distribution. Encyclopedia of Mathematics. Springer Science & Business Media.

[2] Lyon, A. (2014). Why are normal distributions normal? The British Journal for the Philosophy of Science, 65(3), 621-649.

[3] LeCun, Y. A., Bottou, L., Orr, G. B., & Müller, K. R. (2012). Efficient backprop. In Neural networks: Tricks of the trade (pp. 9-48). Springer, Berlin, Heidelberg.

[4] Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).

[5] Wilcox, R. R. (2012). Introduction to robust estimation and hypothesis testing. Academic press.

---

Source: [The Normal Distribution: A Data Scientist‘s Best Friend](https://33rdsquare.com/statistics-data-science-normal-distribution/)