Unveiling the Shape of Data: Skewness, Kurtosis, and CV in AI and Machine Learning

As artificial intelligence (AI) and machine learning (ML) revolutionize industries from healthcare to finance to transportation, the importance of deeply understanding the structure of data has never been greater. AI/ML models are only as good as the data they are trained on, and overlooking key characteristics like the shape of the distribution can lead to suboptimal or even dangerously biased systems.

This is where statistical measures like skewness, kurtosis, and the coefficient of variation (CV) come in. While often overshadowed by more basic metrics like the mean and standard deviation, these measures provide crucial information about the asymmetry, tail extremity, and relative variability of data. For AI/ML practitioners, they are invaluable tools for detecting anomalies, comparing feature distributions, assessing model assumptions, and much more.

In this article, we‘ll take a comprehensive look at skewness, kurtosis, and CV from an AI/ML perspective. We‘ll dive into the technical details of how they are calculated and interpreted, visualize their effects on data distributions, and explore their many applications in real-world AI/ML systems. Whether you‘re a data scientist, researcher, or business leader working with AI/ML, understanding these measures is key to maximizing the power and reliability of your data-driven insights.

Measuring Asymmetry: The Basics of Skewness

Skewness is a measure of the asymmetry of a probability distribution. A symmetric distribution, like the famous bell curve (normal distribution), has zero skewness. But in reality, many data distributions are asymmetric, with the bulk of the data concentrated on one side of the mean.

Mathematically, skewness is defined as:

$ Skewness = \frac{E[(X-\mu)^3]}{\sigma^3} $

where $\mu$ is the mean, $\sigma$ is the standard deviation, and $E$ is the expectation operator. In simpler terms, skewness measures the average cubed deviation from the mean, normalized by the cubed standard deviation.

The sign and magnitude of the skewness value indicate the direction and degree of asymmetry:

• Negative skew (left-skewed): Mean < Median, longer left tail
• Zero skew (symmetric): Mean = Median, equal tails
• Positive skew (right-skewed): Mean > Median, longer right tail

Common guidelines for interpreting skewness values:

• -0.5 to 0.5: Approximately symmetric
• -1 to -0.5 or 0.5 to 1: Moderately skewed
• < -1 or > 1: Highly skewed

Here‘s a visualization of differently skewed distributions:

In AI/ML, skewness is important to assess for several reasons:

1. Many models, like linear regression and LDA, assume normally distributed inputs. Highly skewed features may need to be transformed before modeling.
2. Skewed target variables can bias model training and evaluation metrics. For example, in fraud detection where frauds are rare (high positive skew), accuracy is a misleading metric.
3. Skewness can help identify anomalies or novel examples in the data. In predictive maintenance, extremely positively skewed sensor readings may indicate imminent failure.

Some real-world examples of highly skewed data in AI/ML:

• Word frequencies in natural language processing (NLP) follow a Zipf distribution (high positive skew), with a few common words and many rare words.
• User engagements (clicks, purchases, etc.) on e-commerce platforms are often highly right-skewed, with a "long tail" of low-engagement users.
• Insurance claim amounts are typically right-skewed, with a few very large claims among many smaller ones. This affects risk modeling and pricing.

Quantifying Tail Extremity: The Kurtosis Coefficient

While skewness measures the asymmetry of a distribution, kurtosis measures the thickness of the tails relative to a normal distribution. High kurtosis indicates heavy tails, meaning a higher proportion of extreme values than would be expected under normality.

The kurtosis coefficient is defined as:

$ Kurtosis = \frac{E[(X-\mu)^4]}{\sigma^4} $

where the symbols have the same meanings as in the skewness formula. Kurtosis is essentially the average squared squared deviation from the mean, normalized by the fourth power of the standard deviation.

There are three main types of kurtosis:

1. Mesokurtic: Kurtosis = 3, normal distribution tails
2. Leptokurtic: Kurtosis > 3, heavier than normal tails
3. Platykurtic: Kurtosis < 3, lighter than normal tails

Here‘s a visual comparison:

Excess kurtosis, defined as kurtosis – 3, is often used to shift the baseline to 0 for a normal distribution. Excess kurtosis ranges from -2 to +∞.

In AI/ML, high kurtosis can have significant impacts:

1. Heavy tails mean more outliers, which can bias model training and skew evaluation metrics. Robust methods like median absolute error may be preferred over mean squared error.
2. High kurtosis features may benefit from normalization or Winsorization (capping extreme values) before modeling.
3. In anomaly detection, high kurtosis can make it harder to identify true anomalies among the extreme values. Multivariate methods may be needed.

Some examples of high kurtosis data in AI/ML applications:

• Financial asset returns often have high kurtosis (heavy tails), especially during market crashes. This has major implications for financial risk modeling and portfolio optimization.
• In medical diagnosis from imaging data (e.g. tumor detection), diseased cases often appear as high kurtosis outliers among the healthy majority. Special over-sampling or cost-sensitive learning techniques may be needed.
• Social network degree distributions typically have very high kurtosis, with a few mega-hub nodes among many low-degree nodes. This affects influence modeling and community detection.

Comparing Relative Variability: The Coefficient of Variation

The coefficient of variation (CV) is a normalized measure of dispersion that allows for comparison between variables with different units or scales. It expresses the standard deviation as a percentage of the mean:

$ CV = \frac{\sigma}{\mu} \times 100\% $

where $\sigma$ is the standard deviation and $\mu$ is the mean. The higher the CV, the greater the relative variability in the data.

Some key properties and uses of CV in AI/ML:

• CV is unitless, so it can be used to compare the relative variability of features measured in different units (e.g. height vs. weight).
• CV is often used to assess the quality or consistency of input data. Features with very high CV may need to be discarded or transformed.
• In laboratory medicine, CV is a standard measure of assay precision. AI/ML models for diagnostic prediction need to account for assay variability.
• CV can be used to compare model performance across different data sets or metrics. A model with lower CV predictions is more precise.

Real-world examples of using CV in AI/ML:

• In churn prediction for subscription businesses, CV can be used to compare the relative variability of customer lifetime value (LTV) between customer segments. Higher CV segments may need special retention strategies.
• In sports analytics, CV is often used to assess the consistency of player or team performance statistics over time. Lower CV indicates more reliable performance.
• In traffic flow prediction, CV can compare the relative variability of traffic speeds between different roads or time periods. Higher CV roads may need different prediction models than lower CV roads.

Multivariate Extensions and Recent Research

So far, we‘ve focused on univariate measures of skewness and kurtosis. But in most AI/ML applications, we deal with multivariate data with complex interactions between features. Multivariate skewness and kurtosis measures like Mardia‘s coefficients quantify the joint asymmetry and tail thickness of multivariate distributions.

Mardia‘s multivariate skewness is defined as:

$ b{1,p} = \frac{1}{n^2} \sum{i=1}^n \sum_{j=1}^n [(x_i – \bar{x})‘ S^{-1} (x_j – \bar{x})]^3 $

and Mardia‘s multivariate kurtosis is defined as:

$ b{2,p} = \frac{1}{n} \sum{i=1}^n [(x_i – \bar{x})‘ S^{-1} (x_i – \bar{x})]^2 $

where $x_i$ are the $n$ multivariate observations, $\bar{x}$ is the sample mean vector, and $S$ is the sample covariance matrix.

These multivariate measures are especially important in AI/ML applications like:

• Outlier and novelty detection in high-dimensional data
• Feature selection and dimensionality reduction
• Assessing multivariate normality assumptions of models
• Comparing distributions between multivariate data sets

Recent research has highlighted the importance of skewness, kurtosis, and CV in various AI/ML contexts:

• Zhang et al. (2021) proposed a new feature selection method based on maximizing the skewness of the projected data, which improved classification accuracy on imbalanced data sets.
• Luo et al. (2020) developed a kurtosis-based weighted random forest model that achieved state-of-the-art performance on several anomaly detection benchmarks.
• Rahman and Sarker (2021) used CV to assess the stability of deep learning models for stock price prediction across different market regimes.

As the complexity and stake of AI/ML systems continue to grow, research into leveraging distributional shape for more reliable and robust modeling will only accelerate.

Conclusion

We‘ve taken a deep dive into three key measures of distributional shape: skewness, kurtosis, and coefficient of variation, and explored their many implications and applications in AI and machine learning.

• Skewness measures the asymmetry and direction of a distribution‘s tails relative to its mean. It‘s crucial for assessing normality assumptions, detecting anomalies, and handling imbalanced data.
• Kurtosis measures the thickness of a distribution‘s tails relative to a normal distribution. High kurtosis means more outliers and extreme values, which can bias models and complicate anomaly detection.
• Coefficient of variation (CV) measures relative variability, allowing for comparison across different scales. It‘s useful for assessing data quality, comparing model precision, and analyzing relative consistency.

We‘ve also seen how these concepts extend to the multivariate case and highlighted some recent research on their applications in state-of-the-art AI/ML systems for anomaly detection, feature selection, and model evaluation.

The key takeaway is this: in the era of big data and high-stakes AI, surface-level aggregation is not enough. To build truly robust and insightful AI/ML systems, we must peer deeper into the structure of our data and leverage the rich information contained in distributional shape. Skewness, kurtosis, and CV are three powerful tools for doing just that.

So the next time you‘re exploring a data set or debugging an underperforming model, remember to check these measures. The secrets to better performance may be hiding in the tails!