Feature Engineering: Detecting and Removing Outliers Using Python

Outliers are data points that significantly differ from other observations in a dataset. They can arise from measurement errors, data entry mistakes, or they may be valid data points that are simply unusual cases. In the context of machine learning, outliers can have a major impact on the performance of algorithms, often leading to skewed or inaccurate results if not dealt with appropriately. Therefore, detecting and handling outliers is a crucial step in the feature engineering process when preparing data for machine learning models.

In this article, we will take an in-depth look at various techniques for identifying outliers in datasets using Python. We will cover statistical methods, proximity-based approaches, and clustering-based outlier detection, along with visualizations that help spot outliers. Code examples are provided throughout to demonstrate how to implement these techniques in Python using popular libraries such as scikit-learn and PyOD. We will also discuss best practices around when it makes sense to remove outliers entirely versus keeping them in the data, and strategies like capping or clipping outlier values as an alternative to full removal.

Why Outliers Matter in Machine Learning

Before diving into outlier detection techniques, it‘s important to understand why identifying and handling outliers is so essential in machine learning. Many machine learning algorithms, especially those that rely on distance metrics or statistical properties of the data, are sensitive to outliers. Here are some of the key ways outliers can negatively impact ML models:

• Skewed results: Outliers can significantly pull the mean and variance of a feature in a certain direction, distorting the true distribution and relationships in the data. This can lead to skewed or biased model results.

• Reduced accuracy: Outliers may be given undue influence or weight in model training, causing the model to be fit to noise rather than the true underlying patterns. This often reduces overall accuracy on normal, non-outlier data.

• Longer training times: Outlier points can slow down training of machine learning algorithms like gradient descent that rely on computing gradients across training examples.

• Masked insights: Outliers can hide interesting insights that could otherwise be gleaned from the data. Unusual but valid data points may contain valuable information that gets lost among extreme outlier noise.

On the other hand, blindly removing outliers without proper analysis is also problematic, as it may lead to ignoring useful information and edge cases that could improve model robustness. The goal should be to identify outliers and make an informed decision on handling them based on domain knowledge and the specific problem at hand.

With this context in mind, let‘s look at some effective techniques for uncovering outliers in datasets using Python.

Statistical Outlier Detection Methods

Statistical methods are some of the most common and straightforward approaches for identifying outliers based on the distribution of a feature. These techniques typically measure how far away a given point is from the center of the distribution, flagging points that fall beyond a certain threshold as potential outliers.

Z-score

The z-score (or standard score) measures how many standard deviations a data point is from the mean of a distribution. For a feature x, the z-score of the ith data point is calculated as:

$z_i = \frac{x_i – \mu}{\sigma}$

where $\mu$ is the mean of the feature and $\sigma$ is the standard deviation.

Assuming a Gaussian distribution, a common rule of thumb is that z-scores greater than 3 or less than -3 indicate an outlier, as 99.7% of values fall within 3 standard deviations of the mean.

Here‘s how to find outliers based on z-score using Python and scipy:

from scipy import stats

z = np.abs(stats.zscore(data))
outliers = np.where(z > 3)

A slight modification to make this approach more robust to extreme outliers is to replace the mean and standard deviation with the median and median absolute deviation (MAD) when calculating z-scores.

Interquartile Range (IQR)

The interquartile range (IQR) is another common statistical technique for finding outliers that is less sensitive to extreme values than z-scores. The IQR is the range between the 1st quartile (25th percentile) and the 3rd quartile (75th percentile). Data points falling below Q1 – 1.5 IQR or above Q3 + 1.5 IQR are considered outliers.

To find outliers using the IQR method in Python:

Q1 = np.percentile(data, 25)
Q3 = np.percentile(data, 75)
IQR = Q3 - Q1

outliers = (data < (Q1 - 1.5 * IQR)) | (data > (Q3 + 1.5 * IQR))

Proximity-Based Outlier Detection

Statistical methods work well for univariate outliers in roughly Gaussian distributions, but they may miss outliers in multivariate feature spaces with arbitrary shapes. This is where proximity-based outlier detection comes in. The intuition behind proximity-based methods is that normal data points tend to have nearby neighbors, while outliers are isolated in the feature space.

K-Nearest Neighbors (KNN)

One simple proximity-based approach is to measure the distance of each point to its kth nearest neighbor. Points with the largest k-neighbor distances are considered outliers. This method is available in scikit-learn:

from sklearn.neighbors import NearestNeighbors

nbrs = NearestNeighbors(n_neighbors=k)
nbrs.fit(data)

distances, indices = nbrs.kneighbors(data)
outliers = np.argmax(distances[:,-1])

Local Outlier Factor (LOF)

The local outlier factor (LOF) algorithm compares the local density of a point to the local densities of its neighbors, identifying points that have substantially lower density than their neighbors as outliers. LOF can detect outliers in datasets that have regions of varying density, which KNN struggles with.

To use LOF in Python with PyOD:

from pyod.models.lof import LOF

lof = LOF(n_neighbors=20)
lof.fit(data)

outlier_scores = lof.decision_scores_
outliers = lof.predict(data)

Clustering-Based Outlier Detection

Clustering algorithms group similar data points together. Assuming most of the data is normal, outliers should not belong to any cluster or should belong to small, sparse clusters.

DBSCAN

Density-based spatial clustering of applications with noise (DBSCAN) is a popular clustering algorithm that groups points that are closely packed and marks points in low-density regions as outliers. It takes two parameters: epsilon, which specifies how close points should be to be considered a cluster, and min_samples, the minimum number of points required to form a dense region. Any point not belonging to a cluster is considered an outlier (labeled as -1).

Here‘s how to use DBSCAN for outlier detection with scikit-learn:

from sklearn.cluster import DBSCAN

dbscan = DBSCAN(eps=0.5, min_samples=5)
clusters = dbscan.fit_predict(data)

outliers = np.where(clusters == -1)

Visualizing Outliers

Visualizations of the data can be immensely helpful for spotting outliers, especially during initial data exploration before applying computational detection techniques. Here are a couple key plots for visualizing outliers.

Box Plot

Box plots (or box-and-whisker plots) are a convenient way to visually identify outliers in each feature. The box shows the quartiles of the distribution, and the whiskers extend to the farthest non-outlier points. Any points beyond the whiskers are considered outliers.

import matplotlib.pyplot as plt

plt.boxplot(data)
plt.show()

Scatter Plot

For visualizing outliers across two dimensions, scatter plots are a natural choice. Plotting each data point on two selected features can reveal outliers that fall far outside the main cluster of points.

plt.scatter(data[:, 0], data[:, 1])
plt.show()

Best Practices for Handling Outliers

Understanding the nature and cause of outliers is critical for determining the best way to handle them. In some cases, outliers should be removed to prevent them from unduly influencing models. Other times, they may be valid data points that contain valuable information.

Some key considerations when dealing with outliers:

• Domain knowledge: Consult domain experts to understand whether outliers are likely to be erroneous or legitimately unusual cases. This can inform whether removal is appropriate.

• Capping vs removal: Rather than removing outliers completely, capping (or clipping) their values to some reasonable upper/lower bound can be a good compromise to limit their impact without fully discarding them.

• Separate models: If outliers represent distinct but valid behavior, training separate models for normal cases vs outliers may be warranted.

• Beware of masking: Removing the most extreme outlier can cause the second most extreme point to then be flagged as an outlier. Repeat outlier removal with caution to avoid masking interesting data.

Conclusion

We‘ve covered several powerful techniques for identifying outliers in data using Python, including statistical methods, proximity-based approaches, and clustering-based outlier detection. Proper handling of outliers is a key part of the feature engineering process and can substantially improve the accuracy and reliability of machine learning models. However, it is important to approach outliers thoughtfully and understand the reasons behind them to determine the appropriate treatment. In general, a combination of automated outlier detection methods and manual exploration through visualizations is recommended to get a complete picture of the abnormal points in a dataset. With the right techniques and a bit of domain knowledge, outliers can be transformed from model-damaging noise to model-improving signals.