Comprehensive Guide to Outlier Detection: IQR, Z-score, LOF and DBSCAN Methods

Introduction

Outliers are data points that fall significantly outside the range of the majority of the data. In contrast, inliers are the "normal" data points that align with most of the dataset. Outliers can arise from data entry errors, faulty sensors, rare events, or previously unseen patterns emerging in the data.

Detecting outliers is a crucial part of the data cleaning and preparation process. Including outliers in an analysis can dramatically skew statistical measures like the mean and lead to misleading results. However, outliers are not always invalid data points. In some cases, they can reveal valuable insights about the data generation process or flag important events that require special attention.

There are three main types of outliers:

1. Point or global outliers: Single data points that fall far from the main distribution
2. Collective outliers: Groups of data points that collectively diverge from the dataset, even if individually they appear normal
3. Contextual outliers: Data points that are anomalous only in a specific context, like a 90°F day in Antarctica

This article will dive into four popular methods for outlier detection: Interquartile Range (IQR), Z-score, Local Outlier Factor (LOF), and Density-Based Spatial Clustering of Applications with Noise (DBSCAN). We‘ll explain the intuition and math behind each method, demonstrate how to implement them in Python, compare their strengths and weaknesses, and discuss how to choose the right approach for your data.

To illustrate these techniques, we‘ll use a dataset of scrap metal sales spanning 2018-2022. The dataset contains the scrap type, sale date, weight, and price per kilogram. Our goal is to identify any anomalous prices or weights in the data before conducting further analysis. Let‘s get started!

Interquartile Range (IQR) Method

The Interquartile Range (IQR) measures the spread of the middle 50% of the data. It‘s a robust statistic that is less sensitive to extreme values than the range.

To calculate the IQR:

1. Arrange the data in ascending order
2. Divide the data into four equal parts at the 25th, 50th, and 75th percentiles. These dividing values are called the first quartile (Q1), median, and third quartile (Q3).
3. Calculate the IQR as Q3 – Q1

Values falling below Q1 – 1.5 x IQR or above Q3 + 1.5 x IQR are considered outliers. These boundaries are called the lower and upper fences.

The IQR method works well for skewed distributions since it doesn‘t assume the data follows any particular distribution. It‘s easy to calculate and interpret. However, it struggles with multivariate data and can be distorted by a large number of identical values.

Let‘s apply the IQR method to the scrap metal price data:

import numpy as np

def outliers_iqr(vals):
    q1 = np.percentile(vals, 25)
    q3 = np.percentile(vals, 75)
    iqr = q3 - q1
    lower_fence = q1 - 1.5 * iqr
    upper_fence = q3 + 1.5 * iqr
    return np.where((vals < lower_fence) | (vals > upper_fence))

price_outliers = outliers_iqr(df[‘Rate in Rs./Kg.‘])

print(f"Possible outliers at indices: {price_outliers}")
print(f"Values: {df[‘Rate in Rs./Kg.‘].iloc[price_outliers]}")

This flags prices below Rs. 12.0/kg and above Rs. 45.0/kg as potential outliers. We‘d want to investigate if these are data entry errors or genuinely exceptional prices before proceeding with analysis. The lower outliers in particular seem suspect and may need to be removed or corrected.

Z-score Method

The Z-score, or Standard Score, measures how many standard deviations a data point is from the mean. The formula for a Z-score is:

Z = (x – μ) / σ

Where:

• x is the value of the data point
• μ is the mean of the distribution
• σ is the standard deviation

Z-scores allow us to compare values from different distributions on a common scale. A Z-score tells you if a value is typical for the distribution or not.

The usual rule of thumb is that a Z-score above 3 or below -3 indicates an outlier, as 99.7% of values fall within three standard deviations of the mean in a normal distribution. However, you can adjust this threshold depending on how strictly you want to define an outlier.

The Z-score method assumes the data follow a normal distribution. It can give misleading results for highly skewed data. But it‘s simple to implement and works well for approximately symmetric distributions:

from scipy.stats import zscore

weight_outliers = np.abs(zscore(df[‘Scrape_Qty‘])) > 3

print(f"Possible outliers at indices: {np.where(weight_outliers)}")  
print(f"Values: {df[‘Scrape_Qty‘].iloc[np.where(weight_outliers)]}")

Comparing the distributions of the scrap weights before and after removing these outliers shows that the extreme high values have been clipped, bringing the distribution closer to normal.

Local Outlier Factor (LOF)

The LOF algorithm is a density-based method that identifies outliers in a dataset by comparing the local density of a point to the local densities of its neighbors. It works well for datasets with regions of varying density.

The key concepts in LOF are:

• k-distance: The distance of a point to its kth nearest neighbor
• Reachability distance: The maximum of a point‘s k-distance and the actual distance to a particular neighbor
• Local reachability density (LRD): The inverse of the average reachability distance of a point from its neighbors
• LOF: The average LRD of a point‘s neighbors divided by the point‘s LRD

A point with a much lower density than its neighbors (i.e., higher LOF) is likely to be an outlier. Typically, points with LOF around 1 are normal, while outliers have LOF significantly greater than 1.

The main parameters for LOF are the number of neighbors to consider (k) and the threshold LOF value for declaring a point an outlier. These need to be tuned for each dataset.

Here‘s how to apply LOF to the scrap data using the PyOD library:

from pyod.models.lof import LOF

lof = LOF(n_neighbors=20, contamination=0.02)
outlier_scores = lof.fit_predict(df[[‘Scrape_Qty‘,‘Rate in Rs./Kg.‘]])

df[‘outlier‘] = outlier_scores
print(df[df[‘outlier‘] == 1][[‘Scrape_Qty‘,‘Rate in Rs./Kg.‘]])

This flags the points with the top 2% of LOF scores as outliers. Visualizing the outliers shows they are mostly points falling in low-density regions, even if their individual feature values are not extreme.

The strength of LOF is detecting outliers in heterogeneous datasets where simpler methods may fail. But it is more computationally intensive, requires careful parameter selection, and can struggle with high-dimensional data.

DBSCAN

DBSCAN is another density-based algorithm that finds clusters of points and marks points in low-density regions as outliers. It‘s well-suited for data with irregular cluster shapes and sizes.

DBSCAN works by categorizing points as core points, border points, or noise points:

• Core points have at least MinPts neighboring points within a distance ε
• Border points are neighbors of a core point but don‘t meet the MinPts threshold
• All other points are noise/outliers

Clusters are formed by connecting core points and their borders. Noise points don‘t belong to any cluster.

The key parameters for DBSCAN are:

• ε (eps): Maximum distance for two points to be considered neighbors
• MinPts: Minimum number of points within ε for a point to be a core point

To choose ε, we can look at the k-distance plot, which shows the distance to the kth nearest neighbor for each point, sorted in descending order. A good value for ε is where this plot shows an "elbow".

We can implement DBSCAN for the scrap data as follows:

from sklearn.cluster import DBSCAN 
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
X = scaler.fit_transform(df[[‘Scrape_Qty‘,‘Rate in Rs./Kg.‘]])

dbscan = DBSCAN(eps=0.2, min_samples=20)  
clusters = dbscan.fit_predict(X)

df[‘cluster‘] = clusters
df[‘outlier‘] = clusters == -1

print(f"Total outliers detected: {sum(df[‘outlier‘])}")

With ε=0.2 and MinPts=20, DBSCAN finds several outliers that are spatially separated from the main clusters in the data.

DBSCAN is a powerful algorithm that can find non-linearly separable clusters. But it struggles with varying density clusters, is sensitive to parameter choice, and has trouble with high-dimensional data due to the curse of dimensionality.

Comparing the Methods

Each outlier detection method has its strengths and weaknesses:

• IQR is simple, interpretable and works well for skewed data, but can‘t handle multivariate data
• Z-score is also easy to implement and works for roughly symmetric data, but assumes normality
• LOF can detect outliers in heterogeneous data, but is computationally costly and needs careful parameter tuning
• DBSCAN finds arbitrarily shaped clusters and outliers, but is sensitive to parameters and density differences

The best method depends on your data‘s characteristics and your analysis goals. If you have mostly univariate data, IQR is a good choice. For roughly normal data, Z-score is appropriate. LOF and DBSCAN are better for complex multivariate datasets, with LOF handling varying densities better.

But no matter what method you use, it‘s crucial to bring in domain expertise to validate the detected outliers. Outliers can be real anomalies that deserve special attention. Blindly removing them can erase valuable signals. Understanding the data generating process should guide outlier treatment.

Conclusion

We‘ve seen four key methods for outlier detection and how to apply them in Python. Identifying and handling outliers is a key skill in data science, as they can drastically affect your results if ignored.

The IQR and Z-score methods are simple univariate approaches best suited for unimodal data. LOF and DBSCAN are more advanced algorithms that can find outliers in complex multivariate datasets.

But outlier detection is as much an art as a science. It requires careful thought about the nature of your data and the impact of extreme values on your analysis. Always combine statistical methods with domain knowledge to make the call on whether an outlier is a data error, a random fluctuation, or an informative anomaly.

Hopefully this article has equipped you with the conceptual understanding and practical tools to tackle outliers in your own data. Keep exploring and experimenting – mastering outlier detection will make you a much more effective data scientist. Happy outlier hunting!