Skewness in Statistics: A Deep Dive for Data Scientists

Skewness is a fundamental concept in statistics that measures the asymmetry of a probability distribution. While often overlooked in introductory statistics courses, skewness can have major impacts on data analysis and modeling, especially in the era of big data and machine learning. In this in-depth guide, we‘ll explore skewness from a data science perspective, with a focus on positively skewed distributions.

Defining Skewness

Formally, skewness is the third standardized moment of a distribution, defined as:

$$ \text{Skewness} = \frac{\mathbb{E}\left[(X – \mu)^3\right]}{\sigma^3} $$

where $X$ is a random variable, $\mu$ is the mean, and $\sigma$ is the standard deviation.

In simpler terms, skewness measures how lopsided a distribution is. A symmetric distribution, like the normal distribution, has zero skewness. The left and right tails are perfect mirror images, and the mean equals the median.

A distribution with positive skewness has a longer right tail, with the mean greater than the median. The bulk of the values cluster around the left side, with a few extreme values pulling the mean to the right.

Conversely, a distribution with negative skewness has a longer left tail, with the mean less than the median.

Skewness is closely related to the other moments of a distribution:

• The first moment is the mean, measuring central tendency
• The second moment is the variance, measuring dispersion
• The third moment is skewness, measuring asymmetry
• The fourth moment is kurtosis, measuring the thickness of the tails

Together, these moments provide a complete description of the shape of a distribution.

Measures of Skewness

There are several common measures of skewness, each with different properties:

• Pearson‘s first coefficient of skewness is based on the difference between the mean and the mode, standardized by the standard deviation. It‘s used when the mode is known.

$$ \text{Sk}_\text{Pearson 1} = \frac{\text{Mean} – \text{Mode}}{\text{Standard Deviation}} $$

• Pearson‘s second coefficient of skewness is based on the difference between the mean and the median, standardized by the standard deviation. It‘s used when the mode is unknown.

$$ \text{Sk}_\text{Pearson 2} = 3 \left(\frac{\text{Mean} – \text{Median}}{\text{Standard Deviation}}\right) $$

• The Bowley coefficient of skewness is a quantile-based measure, comparing the sizes of the upper and lower quartiles.

$$ \text{Sk}_\text{Bowley} = \frac{Q_3 + Q_1 – 2Q_2}{Q_3 – Q_1} $$

• Kelly‘s measure of skewness is another quantile-based measure, using the 90th and 10th percentiles.

$$ \text{Sk}\text{Kelly} = \frac{P{90} + P{10} – 2P{50}}{P{90} – P{10}} $$

Each of these measures has different strengths and weaknesses. The Pearson coefficients are sensitive to outliers, while the quantile-based measures are more robust. The choice of measure depends on the nature of the data and the goals of the analysis.

Causes and Examples of Positive Skewness

Positively skewed distributions are very common in real-world data, especially in fields like economics, finance, and social science. Some common causes of positive skewness include:

• Lower bounds: Variables with a natural lower bound but no upper bound, like income, wealth, and prices, tend to be right-skewed. Most values cluster near the lower bound, with a diminishing tail of high values.

• Multiplicative processes: When a quantity grows by random proportions over time, like compound interest or viral spreading, the result is often a lognormal distribution, which is positively skewed.

• Power laws: Phenomena that follow a power law or Pareto distribution, like city populations, earthquake magnitudes, and word frequencies, are characterized by a highly skewed distribution with a few extreme values.

Here are some examples of positively skewed distributions that commonly arise in data science:

• Income distribution: In most countries, the majority of people earn below the mean income, with a small number of very high earners pulling the mean upward. The Gini coefficient, a measure of income inequality, is directly related to the skewness of the income distribution.

• Social media engagement: The number of likes, shares, and comments on social media posts typically follows a highly skewed distribution. A small number of viral posts receive huge engagement, while the vast majority receive little attention.

• Website traffic: The distribution of pageviews across websites is often highly skewed, with a few popular sites receiving the bulk of the traffic. This is an example of a power law distribution.

• Insurance claims: The size of insurance claims is usually positively skewed, with many small claims and a few very large ones. This has important implications for risk management and pricing.

Impact on Statistical Analysis

Skewness can have major consequences for statistical analysis, especially when using methods that assume normality. Here are some of the main problems that arise with skewed data:

• Inflated mean: In a positively skewed distribution, the mean is pulled higher by the extreme values in the right tail. This can give a misleading impression of the typical value in the dataset.

• Inflated variance: The squared deviations from the mean are also influenced by the extreme values, leading to an overestimate of the variance and standard deviation.

• Biased parameter estimates: When fitting models like linear regression to skewed data, the parameter estimates can be biased and inconsistent. The model may be unduly influenced by the extreme values.

• Incorrect confidence intervals and p-values: Many statistical tests and confidence intervals rely on the assumption of normality. With skewed data, these can be misleading and lead to incorrect conclusions.

• Reduced power: Skewness can reduce the power of statistical tests to detect significant differences or relationships, especially with small sample sizes.

To illustrate these issues, let‘s simulate some positively skewed data in Python using the lognormal distribution:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import lognorm

# Parameters for lognormal distribution
mu = 0
sigma = 1

# Generate random sample
data = lognorm.rvs(s=sigma, scale=np.exp(mu), size=1000)

# Plot histogram
plt.hist(data, bins=30, density=True, alpha=0.5)
plt.xlabel(‘Value‘)
plt.ylabel(‘Density‘)
plt.title(‘Lognormal Distribution‘)
plt.show()

# Compute summary statistics
print(f‘Mean: {np.mean(data):.3f}‘)
print(f‘Median: {np.median(data):.3f}‘)
print(f‘Standard Deviation: {np.std(data):.3f}‘)
print(f‘Skewness: {stats.skew(data):.3f}‘)

Output:

Mean: 1.649
Median: 1.000
Standard Deviation: 2.161
Skewness: 6.185

As expected, the mean is much higher than the median due to the right skew, and the standard deviation is inflated. The skewness statistic confirms the strong positive skew.

Dealing with Skewed Data

So what can we do when confronted with skewed data? There are several approaches, each with their own tradeoffs:

1. Transformations: One common approach is to apply a mathematical transformation to the skewed variable to make it more symmetrical. Typical choices are the log transformation, power transformations like the Box-Cox or Yeo-Johnson, or the inverse hyperbolic sine transformation. The choice depends on the specific shape of the distribution and the presence of zero or negative values.

2. Robust methods: Another option is to use statistical methods that are less sensitive to skewness and outliers. For example:

• Robust regression methods like least absolute deviation (LAD) or Huber loss minimize the impact of outliers.
• Quantile regression models the conditional median or other quantiles instead of the mean.
• Nonparametric tests like the Mann-Whitney U or Kruskal-Wallis don‘t assume normality.

3. Resampling: Bootstrapping and permutation tests can provide valid inferences without assumptions about the distribution shape. By resampling from the observed data, they approximate the sampling distribution empirically.

4. Anomaly detection: If the skewness is caused by a small number of genuine anomalies, anomaly detection techniques like isolation forests or local outlier factor can identify and remove them. This should be done cautiously to avoid discarding valuable information.

5. Mixture models: In some cases, the skewness may be due to the presence of distinct subpopulations within the data. Mixture models, like Gaussian mixture models or kernel density estimation, can model these as separate components and provide a better fit.

Here‘s an example of using a log transformation and robust regression in Python with the scikit-learn library:

from sklearn.linear_model import HuberRegressor
from sklearn.preprocessing import FunctionTransformer

# Log transform the skewed variable
transformer = FunctionTransformer(np.log1p, validate=True)
X_log = transformer.transform(X)

# Fit a robust Huber regression
huber = HuberRegressor().fit(X_log, y)

Skewness in Machine Learning

Skewness is also an important consideration in machine learning, especially when dealing with imbalanced datasets. In many real-world problems, like fraud detection or rare disease diagnosis, the classes of interest are highly skewed, with many more negative examples than positive ones.

This class imbalance can cause problems for standard machine learning algorithms:

• Algorithms may optimize for overall accuracy and ignore the minority class
• Evaluation metrics like accuracy can be misleading when classes are imbalanced
• Some algorithms assume balanced class probabilities

There are several techniques for dealing with imbalanced data in machine learning:

• Resampling the dataset to be more balanced, either by oversampling the minority class (e.g. SMOTE) or undersampling the majority class
• Using class weights or stratified sampling to give more importance to the minority class
• Choosing evaluation metrics that are robust to imbalance, like precision, recall, F1 score, and area under the precision-recall curve
• Using algorithms that are less sensitive to imbalance, like decision trees, support vector machines with balanced class weights, or anomaly detection methods

Here‘s an example of using SMOTE to oversample the minority class in an imbalanced dataset with Python and imbalanced-learn:

from imblearn.over_sampling import SMOTE
from sklearn.svm import SVC
from sklearn.metrics import f1_score

# Oversample the minority class with SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X, y)

# Fit an SVM with balanced class weights
svm = SVC(kernel=‘rbf‘, class_weight=‘balanced‘).fit(X_resampled, y_resampled)

# Evaluate with F1 score
y_pred = svm.predict(X_test)
f1 = f1_score(y_test, y_pred)
print(f‘F1 score: {f1:.3f}‘)

Philosophical Considerations

While transforming or resampling skewed data is a common practice, some argue that it can remove important information. After all, the skewness is a real feature of the data generating process, not just a statistical inconvenience.

By trying to force the data into symmetry, we may be discarding valuable insights about the underlying system. The extreme values in the tail, while rare, can be the most consequential and informative data points.

There‘s also a question of interpretability. If we build a model on transformed data, the coefficients and predictions are on the transformed scale, which can be hard to translate back to the original units.

Ultimately, the decision of how to handle skewed data depends on the goals and constraints of the analysis. If the primary aim is to make accurate predictions, transformations and robust methods can be very effective. But if the goal is to understand the mechanisms generating the data, it may be worth preserving the skewness and using methods that can accommodate it.

Conclusion

Skewness is a fundamental property of many real-world datasets, especially in the age of big data. As data scientists, it‘s crucial to recognize skewness, understand its impacts on our analyses, and choose appropriate methods to handle it.

While there‘s no one-size-fits-all solution, the techniques covered in this article – transformations, robust methods, resampling, anomaly detection, and mixture models – provide a powerful toolkit for dealing with skewed data.

At the same time, it‘s important to remember that skewness is not just a nuisance to be eliminated, but a potential source of insight and information. By carefully considering the context and goals of the analysis, we can make informed decisions about when and how to handle skewness.

Hopefully this deep dive has given you a comprehensive understanding of skewness from a data science perspective. Happy analysing!