Dealing with Outliers Using the Z-Score Method: A Comprehensive Guide
Imagine you‘re analyzing a dataset of human heights, and you come across someone who is 10 feet tall! That would certainly give you pause and make you question if that extreme value is real or some kind of error. In data science lingo, we call these extreme values that deviate significantly from the rest of the data distribution “outliers”.
Outliers can wreak havoc on your data analysis by skewing summary statistics, reducing model performance, and leading to incorrect conclusions. Thankfully, statistics gives us tools to systematically identify outliers so we can investigate them and decide how to handle them. In this post, we‘ll take an in-depth look at one of the most common outlier detection methods–the z-score technique–and show you how to implement it in Python.
Understanding Z-Scores
Before we dive into the z-score method for outliers, let‘s make sure we understand what z-scores are. A z-score, also known as a standard score, indicates how many standard deviations a data point is from the mean of the dataset. Mathematically, it‘s calculated as:
z = (x – μ) / σ
Where:
- x is a single raw data point
- μ is the mean of the data
- σ is the standard deviation
Z-scores are a way of standardizing data to the same scale by using the mean as the central reference point. Positive z-scores indicate the raw score is higher than the mean, while negative z-scores indicate the raw score is below the mean.
The standard normal distribution has a mean of 0 and standard deviation of 1. Raw data can be converted to z-scores to fit the standard normal distribution. Z-scores tell you how far away a point is from the mean in terms of standard deviations. For example, a z-score of 2 means the data point is 2 standard deviations above the mean.
This standardization allows you to compare data that have different means and standard deviations. It‘s a core concept in statistics used in hypothesis testing, probability calculations, and more.
Using Z-Scores to Find Outliers
So how can we leverage z-scores to detect outliers in a dataset? The basic idea is that data points with z-scores that are too high or too low are considered unusual and potential outliers. But how high or low is too much?
A commonly used threshold is 3 standard deviations from the mean in either direction. In normally distributed data, 99.7% of values lie within 3 standard deviations of the mean, so values outside this range are often considered outliers.
The general recipe is:
- Calculate the mean and standard deviation of the data
- Calculate the z-score of each data point
- Identify outliers as values with z-scores > 3 or < -3
Let‘s walk through an example in Python to make this concrete. We‘ll use the heights dataset mentioned in the intro.
heights = np.array([65, 68, 72, 69, 71, 67, 63, 64, 66, 120])mean_height = np.mean(heights) sd_height = np.std(heights)
print(f”Mean height: {mean_height:.2f} inches”)
print(f”Standard deviation: {sd_height:.2f} inches”)
This prints out:
Mean height: 74.50 inches Standard deviation: 16.69 inches
Now let‘s calculate the z-scores:
z_scores = (heights - mean_height) / sd_height print(z_scores)
Which gives:
[-0.57, -0.39, -0.15, -0.33, -0.21, -0.45, -0.69, -0.63, -0.51, 2.73]
Finally, we can identify outliers as data points with abs(z) > 3:
outliers = heights[np.abs(z_scores) > 3]
print(f"Outlier values: {outliers}")
The output:
Outlier values: [120]
As expected, the extremely tall 120 inch (10 foot) person is flagged as an outlier!
Advantages and Limitations
The z-score method is simple to understand and implement which makes it a popular choice for outlier detection. Since it‘s based on the standard normal distribution, it works best when your data is approximately normally distributed. This is actually the main limitation of z-scores for outliers – they often don‘t work well if the data is heavily skewed or has a non-normal distribution.
Some other advantages of z-scores:
- Puts data on a standardized scale for easier comparison
- Commonly used across many domains and applications
- Implementable from scratch or with common libraries like scipy and sklearn
- Extendable to multivariate data using techniques like Mahalanobis distance
However, the disadvantages include:
- Relies on normality assumption which real-world data often violates
- Sensitive to sample size, works best with n > 30
- Can miss “masked” outliers if data has multiple modes
- Requires complete data, challenging with missing values
Deciding How to Handle Detected Outliers
So you‘ve used z-scores to flag some unusual values in your data – now what? While it can be tempting to just remove these “problematic points”, that isn‘t always the right approach. Fundamentally, outliers can arise for different reasons:
- Data errors: Outliers that are due to data entry, measurement, or processing mistakes. These should generally be corrected or removed.
- Legitimate but rare values: The outlier is a real data point, just an unusually high or low one. These are often worth keeping.
- Indicates different population: Outlier may suggest there are actually two different groups in the data. For example, adults mixed into a dataset of childrens‘ heights. Here you may want to split the data and analyze separately.
- Novel discovery: Sometimes outliers can point to exciting new findings, like a surprisingly effective drug in a clinical trial. Worth investigating!
So before taking action, it‘s important to dig deeper into the potential causes of the outliers. Plotting the data distribution and checking for data quality issues is a good start. Domain expertise can also help shed light on whether an extreme value is plausible or not.
If you do decide to remove outliers, be sure to document the decision and methodology used. It‘s also good practice to compare analysis with and without the outlier values to see their impact.
Alternatively, you may choose to keep the outliers but reduce their impact, using techniques like:
- Transforming the data to make it more normal (log, Box-Cox, etc.)
- Using robust statistical methods that are resistant to outliers (median, trimmed mean, etc.)
- Discretizing continuous data into bins/categories
- Applying algorithms that are less sensitive to outliers (decision trees vs regression)
Multivariate Outliers and Other Methods
While we‘ve focused on univariate outliers so far, some of the most interesting outliers are multivariate, meaning they are unusual combinations of multiple variables. For example, a 5 year old who is 6 feet tall would be a multivariate height-age outlier, even though their individual height and age values may be within normal ranges.
To detect multivariate outliers, we can scale up the z-score approach by calculating the Mahalanobis distance. This measures the distance between a data point and the center of the multivariate distribution, taking into account the covariance structure between variables. Points with a large Mahalanobis distance are considered outliers.
Other multivariate outlier methods include:
- Local Outlier Factor (LOF): Compares local density of a point vs its neighbors
- Isolation Forest: Builds trees that isolate outliers near the root
- DBSCAN clustering: Identifies outliers as points in low-density regions
When it comes to univariate outliers, there are also alternatives to z-scores like:
- IQR method: Flags points < Q1 – 1.5IQR or > Q3 + 1.5IQR as outliers
- Tukey fences: Similar to IQR but uses 1.5 and 3 as thresholds for “possible” and “probable” outliers
- Percentiles: Identify outliers as points < 1st or > 99th percentile
- Grubbs‘ test: Used to detect a single outlier in a univariate dataset that follows normal distribution
Case Study: Detecting Outliers in Exam Scores
Let‘s close with a real-world example of using z-scores for outlier detection. An teacher wants to identify students with unusually low and high scores on a math exam to understand the performance range of the class.
import numpy as np from scipy import stats
scores = [85, 92, 76, 88, 58, 90, 95, 30, 48, 82]
z = np.abs(stats.zscore(scores)) print(z)
This gives:
[0.54, 1.13, 0.05, 0.76, 0.89, 0.94, 1.35, 2.31, 1.55, 0.35]
Using a threshold of 2, the z-scores suggest that the scores of 30 and possibly 48 are low outliers. Plotting the distribution confirms the data is approximately normal with those two low values.
The teacher can now reach out to those students to understand why they struggled and provide support. Without this outlier check, those students may have been missed. The teacher can also rest assured that the high scores, while excellent, are not so extreme as to indicate cheating.
This case study illustrates how outlier detection can provide insights to guide interventions and improve outcomes, even with a small dataset.
Conclusion
In this post, we took a deep dive into identifying outliers with the z-score method. We covered:
- What z-scores are and how to calculate them
- How to use z-scores to find outliers based on a threshold like |z| > 3
- Python code to implement z-score outlier detection
- Advantages and limitations of the z-score method
- Strategies for investigating outlier causes and impacts
- Handling outliers by removing or reducing influence
- Multivariate outlier detection and alternative methods
Here are the key takeaways:
- Outliers can have significant effects on data analysis and it‘s important to identify them
- Z-scores standardize data and can flag values that are too many standard deviations from the mean
- Z-score method works best for unimodal data that is approximately normal
- Root causes of outliers should be investigated before deciding to exclude them
- There are many other univariate and multivariate outlier detection techniques to consider
Outliers can be a tricky part of data analysis, but armed with the right statistical tools, you can tame them. The z-score method is a great place to start and can be supplemented with other techniques as needed. The most important things are to be thoughtful about identifying outliers, thorough in investigating them, and transparent in how you handle them.
By following this framework, you‘ll be able to draw more robust insights from your data that lead to better decisions. Happy outlier hunting!
