Advance Statistics Concepts Every Data Scientist Should Know

Introduction

Hello aspiring data scientist! In previous posts, we covered the essential statistics concepts every data science newbie should know, as well as some intermediate topics to level up your skills. Now, we‘re going to dive even deeper into some advanced statistics that will really boost your data science game. Buckle up!

Chebyshev‘s Inequality

You probably remember that for a normally distributed variable, about 68% of values fall within 1 standard deviation of the mean, 95% within 2 standard deviations, and 99.7% within 3. But what if your data isn‘t normally distributed? That‘s where Chebyshev‘s Inequality comes in.

Chebyshev‘s Inequality states that for any probability distribution, the fraction of values that fall within k standard deviations of the mean is at least 1 – 1/k^2. So even if your data looks nothing like a bell curve, you can still make some guarantees about the spread.

For example, let‘s say you have a variable with a mean of 100 and standard deviation of 15. Chebyshev‘s Inequality says that at least 1 – 1/2^2 = 75% of values will fall between 100 – 215 = 70 and 100 + 215 = 130. And at least 1 – 1/3^2 = 89% will fall between 55 and 145. The bounds aren‘t as tight as for a normal distribution, but it‘s still very useful!

import numpy as np

data = [80, 95, 100, 110, 115, 130]  
mean = np.mean(data)  # 105
std = np.std(data)    # 15.8

print(f‘At least {100*(1 - 1/2**2)}% between {mean-2*std:.1f} and {mean+2*std:.1f}‘)
print(f‘At least {100*(1 - 1/3**2)}% between {mean-3*std:.1f} and {mean+3*std:.1f}‘)

Quantile-Quantile (Q-Q) Plots

A Q-Q plot is a graphical method to compare two probability distributions by plotting their quantiles against each other. It‘s often used to check if a distribution is approximately normal.

To make a Q-Q plot, sort your data and calculate percentiles from 1 to 100. Then plot these versus the same percentiles of a normal distribution. If the distributions match, the points will lie on the line y=x.

If the points curve away from the line, that indicates the shapes of the distributions differ. Points above the line on the right mean your data is right-skewed compared to normal, for example.

import statsmodels.api as sm
import matplotlib.pyplot as plt

sm.qqplot(data, line=‘45‘)
plt.show()

One limitation is that Q-Q plots can be hard to interpret with small sample sizes. So make sure you have sufficient data points to really see the distribution shape.

Bernoulli and Log-Normal Distributions

Besides the famous normal distribution, there are many other probability distributions worth knowing. The Bernoulli distribution models a single binary outcome, like a coin flip. It has a probability p of success, and probability q = 1-p of failure.

The mean of a Bernoulli distribution is simply p. The median is 0 if p < 0.5 and 1 if p > 0.5. If p = 0.5 exactly, the median is 0.5. The variance is pq and the standard deviation is √(pq).

For example, let‘s say you survey a class and find 75% of students enjoy giving presentations. Modeling this as a Bernoulli random variable, p = 0.75. The mean enjoyment is 0.75, the median is 1 (enjoy), the variance is 0.75*0.25 = 0.1875, and the standard deviation is √0.1875 = 0.433.

Another interesting distribution is the log-normal. If the log of a variable is normally distributed, the variable itself follows a log-normal distribution. This often arises with financial variables like income or stock prices.

Log-normal distributions are always right-skewed, with a long tail on the upper end. One way to analyze them is to take the log of your data points to get a normal distribution, then convert results back at the end by exponentiating.

Power Law and Pareto Distributions

Some variables follow a power law, where one quantity varies as a power of the other. This leads to distributions like the Pareto, where a small fraction of the population accounts for a large fraction of the total.

Classic examples are wealth (a small percent own a large percent of the wealth), city populations (a few cities have huge populations), and web traffic (a handful of sites get most of the traffic). This is sometimes called the 80-20 rule – 80% of the result comes from 20% of the instances.

The Pareto distribution has a shape parameter α that controls how heavy the tail is. Larger α means a steeper initial slope and more mass in the tail. It‘s easy to mistake Pareto and log-normal distributions, so be careful!

One clue is that Pareto distributions have a heavier tail, and the log of a Pareto variable will still be right-skewed, while the log of a log-normal becomes normally distributed. When in doubt, do some distribution fitting to see which matches your data better.

Box-Cox Transformation

If you determine your data follows a power law or Pareto distribution, it can be hard to work with because of the skew and heavy tail. One option is to transform it to make it approximately normal. The go-to method is the Box-Cox transformation.

The Box-Cox formula is:

• (x^λ – 1) / λ if λ ≠ 0
• log(x) if λ = 0

The transformation has a parameter λ that controls the strength of the transformation. The optimal λ is usually found by maximum likelihood estimation.

from scipy.stats import boxcox

data_trans, lambda_opt = boxcox(data)
print(f‘Optimal lambda: {lambda_opt:.3f}‘)

After transforming with the optimal λ, your data should be much closer to normal, which makes it easier to analyze with tools expecting normality. Just remember to apply the inverse transformation at the end to get results back on the original scale!

The Box-Cox method only works with positive data, so if you have zero or negative values, you‘ll need to shift them first by adding a constant. There are also alternatives like the Yeo-Johnson transformation that don‘t have this restriction.

Conclusion

We‘ve covered a lot of ground in this post, from Chebyshev‘s Inequality to Box-Cox transformations. I hope these concepts give you a richer understanding of the vast world of probability distributions and some tools to tackle non-normal data.

Don‘t worry if you didn‘t absorb it all the first time. The best way to internalize these ideas is to practice with real datasets. Plot the distributions, test the assumptions, and apply the methods. Over time it will become second nature!

Of course, there are many more statistical concepts we didn‘t have space for here. Entire books have been written on the topic! But you‘re now well equipped to venture out and learn more. Always remember, a data scientist is never done learning. Stay curious, keep asking questions, and don‘t be afraid of the math. Happy analyzing!