Why You Shouldn‘t Just Delete Outliers: An AI/ML Perspective

Outliers, those pesky data points that deviate significantly from the norm, can be a major headache for data scientists and ML engineers. It‘s tempting to simply remove them and move on. However, as an artificial intelligence and machine learning expert, I‘m here to tell you that blindly deleting outliers is often a big mistake. Outliers are more than just annoying anomalies – they can provide critical insights that improve your models and lead to pivotal discoveries. In this post, we‘ll dive deep into the world of outliers, exploring techniques for identifying them, understanding their impact, and strategically handling them to make your models more accurate and robust.

Identifying Outliers: More Than Meets the Eye

Before we get into why you shouldn‘t just delete outliers, let‘s discuss how to find them in the first place. While outliers may seem obvious, there‘s more nuance to outlier detection than you might think.

One common method is to use Z-scores, which measure how many standard deviations a data point is from the mean. Typically, a Z-score greater than 3 or less than -3 indicates an outlier. However, Z-scores assume the data follows a normal distribution, which isn‘t always the case.

For skewed distributions, the interquartile range (IQR) method is more suitable. Points that fall below Q1 – 1.5 x IQR or above Q3 + 1.5 x IQR are considered outliers. The IQR is less affected by extreme values than Z-scores.

But univariate methods like Z-scores and IQR have limitations. They can‘t identify multivariate outliers that are only anomalous when considering multiple features together. In an age-height dataset, a 5‘6" 12-year-old wouldn‘t be a height or age outlier separately. But that combination of height and age would be unusual.

This is where AI/ML techniques like DBSCAN clustering and isolation forests shine. DBSCAN groups points that are closely packed and identifies points in low-density regions as outliers. Isolation forests recursively partition data points based on feature values. Since outliers have feature values far from the norm, they tend to be isolated in fewer partitions.

*DBSCAN identifies outliers as points in low-density regions. Source: [Wikipedia](https://en.wikipedia.org/wiki/DBSCAN)*

It‘s also important to consider contextual outliers – points that are anomalous in a specific context but not globally. A $200 electric bill might not be an outlier in general. But for a 1-bedroom apartment in the summer, it would be unusually high. Techniques like autoencoders and bayesian networks can model the multidimensional relationships that define different contexts.

The takeaway is that outlier detection is complex. Relying on simplistic univariate methods risks missing important anomalies. Using a variety of statistical and ML techniques to identify different types of outliers is crucial.

The Risks of Reckless Outlier Removal

So you‘ve found some outliers in your data. Time to nuke ‘em, right? Not so fast. Blindly removing outliers without careful consideration can do more harm than good.

Consider this real-world example. In the 1980s, researchers at NASA‘s Ames Research Center were studying the ozone layer using data from satellite sensors. They noticed some unusually low ozone readings over Antarctica that were far outside the typical range. Initially, these outliers were removed, as they were assumed to be sensor errors.

However, upon further investigation, those "faulty" readings turned out to be real evidence of a gaping hole in the ozone layer. Had scientists continued to ignore these outliers, we may not have discovered the growing ozone problem until much later. The Montreal Protocol, which phased out ozone-depleting substances, could have been delayed, exacerbating the damage.

*The Antarctic ozone hole, initially dismissed as outlier sensor errors. Source: [NASA](https://ozonewatch.gsfc.nasa.gov/facts/hole_SH.html)*

This illustrates a key risk of reckless outlier removal – losing critical signal in the noise. What may seem like data points that are safely ignorable can actually be canaries in the coal mine for major problems or opportunities.

Removing outliers also risks distorting the true shape of your data. Real-world data often doesn‘t follow a perfect normal distribution. It may be skewed, multimodal, or heavy-tailed. Lopping off the tails to make the data look cleaner can give a misleading picture and hide important characteristics.

Another danger is that consecutive outlier removal can mask its own impact. Say you remove all data points with Z-score > 3. Then you recompute the mean and standard deviation and remove points with the new Z-scores > 3. If you repeat this process, you can end up recursively deleting points until the data is unrecognizably distorted.

*Recursive outlier removal can distort data distributions. Adapted from [Leys et al., 2019](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6709872/)*

Even outliers that stem from data entry or measurement errors can be informative. They may reveal issues with your data collection process that need to be fixed. Silently deleting them can allow those problems to continue unnoticed.

The key point is that outliers are often more than just noise to be filtered out. They can provide incredibly valuable information if handled strategically. Removing them without careful thought risks losing important insights and skewing your results.

Outliers as Features: Embracing the Anomalies

In the world of AI and ML, we often think of outliers as a nuisance that interferes with model training. But what if we flip the script? Instead of trying to remove outliers, we can embrace them as informative features.

Let‘s consider fraud detection as an example. In a dataset of credit card transactions, fraudulent purchases are outliers. They deviate strongly from the typical spending patterns. A simplistic approach would be to delete those outliers so the model can focus on normal transactions.

But that would be a huge mistake! In fraud detection, outliers are exactly what you‘re trying to identify. By including fraudulent outliers in the training data, the model can learn the signatures of fraud and be better equipped to spot it in the future. The outliers are the most valuable data points in this case.

Outliers can also act as edge cases that make models more robust. Say you‘re building a self-driving car AI. If your training data only includes typical highway driving, the model may not know how to handle unusual situations like a mattress falling off a truck in front of it. By purposely including those oddball outlier scenarios in training, you can create an AI that‘s prepared for a wider range of real-world curveballs.

Even outliers that aren‘t the main prediction target can be valuable features. Imagine you‘re modeling customer churn for a subscription service. You notice some outlier customers who use the product far more than typical users. While not directly predicting churn, those power user outliers could be important features. Their abnormally high usage may indicate they‘re getting a lot of value and are less likely to churn.

Outlier-Resistant AI Techniques

Of course, not all AI/ML techniques are equally affected by outliers. Some are more resistant to extreme values than others.

Linear regression, for instance, is easily swayed by outliers. A single extreme point can dramatically change the slope of the best-fit line, skewing the predictions for all other points. Logistic regression and SVMs with certain kernels are similarly sensitive.

*A single outlier can substantially change the slope of a linear regression fit. Adapted from [Towards Data Science](https://towardsdatascience.com/detecting-outliers-in-linear-regression-cfe1dc2aa42e)*

Decision trees and random forests, on the other hand, are less affected by outliers. Since they make splits based on information gain, outliers often end up isolated in their own leaves without impacting other splits. Ensemble methods like bagging and boosting can also help mitigate the influence of outliers.

In deep learning, some architectures are more robust to outliers than others. Convolutional neural networks (CNNs) with max pooling layers are relatively resistant, as a single extreme value gets diluted by the local neighborhood pooling. Autoencoder networks trained to reconstruct normal data can also help identify anomalies as points with high reconstruction error.

*Autoencoders trained on normal data can identify outliers as points with high reconstruction error. Source: [Towards Data Science](https://towardsdatascience.com/anomaly-detection-with-autoencoder-in-pytorch-5a7d894b8534)*

The lesson here is that the impact of outliers depends on the ML technique you‘re using. Understanding which methods are more outlier-sensitive can help you choose a approach that‘s well-suited for your data and problem.

Real-World Outlier Insights

To hammer home the importance of thoughtful outlier handling, let‘s look at some more real-world examples where outliers led to breakthrough insights:

• Cosmic microwave background radiation: In 1965, Arno Penzias and Robert Wilson were using a radio telescope to map signals from the Milky Way. They noticed a persistent background noise that was higher than expected. Rather than discarding it as erroneous, they investigated further. It turned out to be the first observation of cosmic microwave background radiation, a key piece of evidence for the Big Bang theory. Penzias and Wilson won the Nobel Prize for this discovery.

• Netflix prize contamination: In 2006, Netflix held an open competition to improve its recommendation algorithm. They released an anonymized dataset of user ratings for algorithm training. Some users noticed that a few outlier accounts had rating patterns that closely matched popular external rating lists posted online. This revealed that the dataset had been contaminated, with some accounts likely being improperly influenced by the external lists. Identifying this problem was crucial for maintaining the integrity of the competition.

• 2008 financial crisis: Leading up to the 2008 financial meltdown, there were outlier signs of growing risk in the system. Extreme default rates in subprime mortgages, unusually high leverage ratios, and spikes in credit default swaps pointed to instability. But most models used by banks and regulators considered these conditions outliers and excluded them. In hindsight, heeding these outlier warnings could potentially have mitigated the severity of the crisis.

These examples illustrate how outliers can serve as early warning signals for major events and reveal previously unknown phenomena. Dismissing them as mere noise can mean missing out on game-changing discoveries and insights.

The Importance of Outliers for AI Safety

As artificial intelligence systems become more advanced and widely deployed, ensuring their safety and robustness is increasingly critical. Outliers play an important role in this context.

AI systems that are brittle to outliers can fail catastrophically in edge case scenarios. A self-driving car that‘s only trained on typical road conditions may freeze or swerve dangerously when confronted with an unusual situation like a horse running across the highway. An AI medical diagnosis system that ignores outlier symptoms could miss rare but serious diseases.

By incorporating outliers in AI development and testing, we can create systems that gracefully handle the unexpected. Techniques like stress testing with extreme value simulations, adversarial example generation, and anomaly detection can help identify AI blind spots before they cause real-world harm.

Outliers are also crucial for monitoring AI systems for potential malfunction or misuse. An AI content moderation system that starts allowing extreme hate speech could be an outlier indicating a major problem. Unusual patterns of AI-driven trading in financial markets could point to rogue algorithms or attempts at market manipulation.

As AI becomes more powerful, paying close attention to outliers will be essential for ensuring these systems remain safe and beneficial. Dismissing or deleting anomalies risks letting critical problems go unchecked until they cause major damage.

Conclusion

Outliers are often seen as the bane of data scientists‘ existence. It‘s understandable to want to get rid of them and focus on the "normal" data. But as we‘ve seen, simply deleting outliers is a dangerous game.

Outliers can be incredibly informative, revealing everything from data quality issues to groundbreaking new phenomena. In AI and ML, they serve as valuable edge cases that make models more robust and adaptable. Identifying and analyzing outliers is also crucial for AI safety, ensuring these powerful systems can handle the unexpected and are monitored for potential misuse.

So next time you spot an oddball data point, resist the urge to nuke it from orbit. Take the time to investigate it thoughtfully. Use diverse statistical and ML techniques to identify different types of outliers. Consider the context and potential implications. And most importantly, approach outliers with curiosity rather than disdain.

Embracing the outliers may be messier than ignoring them, but it‘s essential for rigorous, insightful, and responsible data science and AI. The road to breakthrough insights and robust intelligence is paved with anomalies. Don‘t be afraid to venture down it.