Detecting Time Series Anomalies with anomalize: A Wikipedia Case Study

As an AI/ML expert, a critical part of your job is to monitor key metrics, spot unusual patterns, and investigate root causes. With time series data, this means finding anomalies—data points that deviate significantly from the norm. Manual anomaly detection isn‘t feasible at scale, so algorithmic approaches are essential.

In this in-depth case study, we‘ll explore a powerful technique for detecting time series anomalies using the anomalize package in R. We‘ll cover the fundamentals of anomaly detection, walk through a realistic example with Wikipedia page view data, and discuss advanced considerations for real-world applications.

Fundamentals of Time Series Anomaly Detection

Time series anomalies can take many forms—sudden spikes or dips, changes in seasonality, gradual trend shifts. Causes range from data quality issues and technical failures to major events and emerging behaviors.

Some common techniques for detecting anomalies in time series:

• Statistical methods: Model the expected behavior and flag deviations exceeding a threshold. Includes averaging, ARIMA, Holt-Winters, etc.
• Machine learning: Train a model like a classifier or autoencoder on normal data and score new data points. More complex but handles non-linear patterns.
• Deep learning: Apply neural networks (CNN, LSTM) to learn complex anomaly types from big data. Computationally intensive but state of the art.

For this analysis we‘ll focus on a statistical approach using the anomalize package, which enables a robust and intuitive decomposition-based workflow in R.

Decomposition-Based Anomaly Detection with anomalize

The core idea of anomalize is to decompose a time series into three components—trend, seasonality, and remainder—then identify outliers in the remainder [1]. It follows these steps:

1. Decompose the time series with STL (Seasonal and Trend decomposition using LOESS)
2. Calculate an expected range of ‘normal‘ values for the remainder
3. Flag data points outside this range as anomalies
4. Recompose the components with anomalies flagged

Key parameters to consider:

• Trend and seasonality windows: Longer windows mean smoother components and fewer anomalies. Set based on data characteristics and domain knowledge.
• Anomaly sensitivity (alpha): Width of ‘normal‘ range, as a percentile threshold. Smaller alpha means more anomalies detected.

Selecting good parameters is crucial for meaningful results. Let‘s see it in action with a case study.

Case Study: Wikipedia Page View Anomalies

Data Loading and Preparation

We‘ll analyze daily page views to English Wikipedia articles in the ‘Random Article‘ category from 2015-2022, obtained from the Wikimedia REST API.

Our data is in a CSV with columns for timestamp, Wikipedia project (e.g. ‘en‘), page name, and views. To prepare it for anomalize, we‘ll:

1. Load libraries and data
2. Convert timestamp to date
3. Filter to English Wikipedia
4. Create a tibble time series (tbl_ts)

library(tidyverse)
library(anomalize)

df <- read_csv("random_article_pageviews.csv") %>%
  rename(date = timestamp, views = views_random_article_normalized) %>% 
  mutate(date = as_date(date)) %>%
  filter(project == "en") %>%
  as_tbl_time(index = date)

glimpse(df)

Rows: 2,660
Columns: 4
$ date    <date> 
$ project <chr>  
$ article <chr>  
$ views   <dbl>  

We now have a tsibble with 2660 daily observations ready for exploration and anomaly detection.

EDA and Decomposition

Let‘s visualize our data to understand patterns and choose initial anomalize parameters.

df %>% autoplot(views) +
  ggtitle("Daily Views of Random Articles on English Wikipedia")

We can see an overall upward trend and strong weekly seasonality (viewership higher on weekdays vs weekends). There‘s also a lot of noise, with many single-day spikes and dips.

Next we‘ll use anomalize‘s time_decompose() to split the series into trend, season, and remainder. We set the trend window to "90 days" and let seasonality default to "auto" (which detects the 7-day cycle).

df_decomposed <- df %>%
  time_decompose(views, trend = "90 days")

df_decomposed %>%
  plot_anomaly_decomposition() +
  ggtitle("Wikipedia Random Article Page Views Decomposition")  

The top panel shows the original data (gray) and smoothed trend (blue). The middle panel isolates the weekly seasonal component. The bottom panel contains the remainder—variation not explained by trend or seasonality—where anomalize looks for outliers.

Detecting and Examining Anomalies

With data decomposed, we‘re ready to find anomalies with anomalize(). Since we have weekly seasonality, we‘ll set frequency to "7 day". We‘ll start with the default alpha of 0.05, which flags about 5% of data as anomalies.

df_anomalized <- df_decomposed %>%
  anomalize(remainder, frequency = "7 day") %>%
  time_recompose()

df_anomalized %>%
  plot_anomaly_decomposition() +
  ggtitle("Anomalized Wikipedia Page Views")

The bands (gray) show the ‘normal‘ range and red dots indicate anomalies. We can see many outliers on the high and low side. Let‘s investigate the top anomalies:

df_anomalized %>%
  filter(anomaly == "Yes") %>%
  arrange(desc(abs(remainder))) %>%
  select(date, views, remainder, anomaly) %>%
  head(10)

        date   views remainder anomaly
1 2022-02-24  253191     40598     Yes 
2 2022-03-19  130521    -72880     Yes
3 2022-03-23  123230    -60714     Yes
4 2022-03-21  125335    -58902     Yes

The largest anomaly was a huge spike on 2022-02-24, the day Russia invaded Ukraine. The next biggest anomalies are drops around 2022-03-21. With more data on current events, site issues, etc, we could dig deeper into potential causes.

We can also tune the alpha parameter to adjust anomaly sensitivity:

df_anomalized_a01 <- df_decomposed %>% 
  anomalize(remainder, frequency = "7 day", alpha = 0.01) %>%
  time_recompose()

df_anomalized_a01 %>%
  plot_anomalies(time_recomposed = TRUE) +
  ggtitle("Wikipedia Anomalies with alpha = 0.01")

Compared to alpha = 0.05, setting alpha = 0.01 significantly narrows the band of ‘normal‘, flagging more data points as anomalies.

Let‘s compare anomaly counts:

df_anomalized %>% 
  summarise(pct_anomalies = mean(anomaly == "Yes"))

df_anomalized_a01 %>%
  summarise(pct_anomalies = mean(anomaly == "Yes"))  

# A tibble: 1 x 1
  pct_anomalies
          <dbl>
1        0.0519

# A tibble: 1 x 1
  pct_anomalies
          <dbl>
1         0.153

With alpha = 0.05, about 5% of data is flagged as anomalous. Reducing to alpha = 0.01 increases it to over 15%! The optimal setting depends on your specific data and how sensitive you want anomaly detection to be.

To evaluate performance, we‘d ideally have labels indicating true anomalies to compare against anomalize‘s output. Useful metrics include precision (what % of detected anomalies are true anomalies), recall (what % of true anomalies were detected), and F1 score (a balance of precision and recall) [4]. Without labels, we can qualitatively assess the anomaly examples and trends to judge if the selected parameters produce reasonable results for the application.

Advanced Anomaly Detection Techniques

While statistical methods like anomalize are useful, they have limitations. They assume linear relationships, require specifying trend and seasonality, and can‘t detect certain complex anomaly types.

Machine learning offers more flexible, data-driven approaches. Density-based models like Isolation Forests and clustering, and reconstruction-based models like PCA and autoencoders, can model complex ‘normal‘ states and detect high-dimensional anomalies with less manual feature engineering [5]. Libraries like scikit-learn and PyOD enable anomaly detection on tabular data.

To scale anomaly detection on large time series datasets, deep learning techniques are increasingly used [6]. Recurrent neural networks (RNN), such as LSTMs, can model complex temporal dependencies and detect subtle anomalies across multiple series. Convolutional neural networks (CNN) apply ‘filters‘ to automatically learn relevant patterns at different temporal scales.

Deep learning approaches require a lot of data and compute resources but are state of the art for complex anomaly detection tasks. Libraries like TensorFlow, PyTorch, and Keras streamline development.

Anomaly Detection in Practice

In a real-world AI/ML system, anomaly detection would be part of a larger workflow [7]:

1. Ingest data from various sources
2. Preprocess and store in a time series database
3. Detect anomalies with scheduled jobs or real-time streaming
4. Store and display results in a monitoring dashboard
5. Alert relevant stakeholders to investigate
6. Provide feedback to tune anomaly detection over time

Beyond univariate anomaly detection on a single series, advanced applications might include:

• Correlating anomalies across multiple related metrics to identify root causes
• Detecting anomalies in multivariate series like sensor data from IoT devices
• Updating forecasting models in response to detected anomalies
• Preventing model drift by monitoring live vs training data distribution

Implementing these would require integration with data pipelines, cloud services, BI tools, and notification channels. End-to-end platforms like AWS Sagemaker and GCP Vertex AI can help manage the model lifecycle.

Conclusions and Resources

Anomaly detection is a critical capability for monitoring time series data at scale. The anomalize package provides a powerful, flexible statistical approach based on time series decomposition and outlier detection in R.

In this analysis, we used anomalize to detect anomalies in Wikipedia page view data. We demonstrated the core workflow—decomposition, anomalization, recomposition, visualization—and explored the impact of tuning trend, seasonality, and sensitivity parameters. Key takeaways:

• Anomaly detection results are highly dependent on decomposition and anomalization parameters. Choose carefully and iterate!
• Collaborate with subject matter experts to validate and investigate anomalies surfaced. Not all statistical anomalies will be relevant to the business.
• Consider the pros and cons of decomposition-based methods vs machine learning and deep learning approaches as data scales and complexities grow.

To learn more about anomaly detection, check out these resources:

• anomalize package and documentation
• Introduction to Anomaly Detection (Anodot blog)
• Anomaly Detection: A Survey (Chandola, Banerjee & Kumar, ACM 2009)
• Machine learning for anomaly detection (Statsbot blog)
• Time Series Anomaly Detection with TensorFlow (TensorFlow blog)

I hope this deep dive equipped you with the foundations and practical steps for applying anomaly detection to uncover insights in your own time series data. Feel free to reach out with any questions!