# Detecting Anomalies in Time Series Data with Facebook Prophet

- Canonical: https://33rdsquare.com/anomaly-detection-model-using-facebook-prophet/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Time series data, or data collected at regular intervals over time, is extremely common across many domains such as finance, IoT sensors, web traffic monitoring, and more. Being able to model time series data to forecast future values and detect unusual patterns or anomalies is a critical skill for data scientists and analysts.

In this post, we‘ll take a deep dive into anomaly detection for time series data using the popular open source library Facebook Prophet. We‘ll explain the different types of anomalies, see how Prophet models time series, and walk through a full example in Python of using Prophet to detect anomalies in taxi ridership data.

## The What and Why of Anomaly Detection

An anomaly, or outlier, is a data point that differs significantly from other observations. In time series data, anomalies can take a couple different forms:

**Global outliers** are individual data points that lie far outside the overall distribution of the time series. For example, a random spike or dip caused by a data collection error. Global outliers can often be spotted visually in a time series plot.

**Contextual anomalies** are data points that look normal compared to the entire dataset, but are unusual compared to the data points around it in the time series. For example, low electricity usage is normal at night, but would be anomalous during the middle of the day. Contextual anomalies take into account the seasonal patterns in the data.

Detecting both global outliers and contextual anomalies is important for several reasons:

1. Identifying data quality issues – Anomalies are often caused by failures in data collection, aggregation, or processing. Catching these early is key.
2. Monitoring systems for unusual behavior – Anomalies in metrics like latency, error rates, or resource utilization can be early warning signs of system failures.
3. Detecting opportunities or threats – Anomalous patterns in business data like product sales or website traffic can signal emerging trends or problems to investigate.
4. Improving forecasting accuracy – Feeding anomalous data points into forecasting models can throw off the results. Identifying and removing anomalies leads to better predictions.

While it‘s possible to find some global outliers through manual visualization and inspection of the data, catching contextual anomalies in particular requires more sophisticated techniques like Prophet that model the underlying patterns in the time series.

## Introducing Facebook Prophet

Prophet is an open source library for time series forecasting developed by Facebook. It‘s designed to be intuitive and easy to use even without extensive experience with time series models.

Under the hood, Prophet is an additive regression model that fits non-linear trends with yearly, weekly, and daily seasonality, plus holiday effects. It works especially well with time series that have strong seasonal effects and several seasons of historical data.

Prophet also has built-in support for detecting anomalous data points that do not fit the overall trend and seasonal patterns. This makes it a great choice for anomaly detection on top of forecasting.

## Example: NYC Taxi Data

To see Prophet in action for anomaly detection, let‘s walk through an example using a real dataset of New York City taxi ridership. The data contains a time series of the number of taxi trips taken each half hour from July 2014 to January 2015.

Our goal will be to model this time series data with Prophet, use it to find any anomalous data points, and visualize the results.

### Loading and Preparing the Data

First we‘ll load the required libraries and read in the taxi data CSV file:

```
import pandas as pd
from fbprophet import Prophet

df = pd.read_csv(‘nyc_taxi.csv‘)
print(df.head())
```

This prints:

```
             timestamp    value
0  2014-07-01 00:00:00  10844.0
1  2014-07-01 00:30:00   8127.0
2  2014-07-01 01:00:00   6210.0
3  2014-07-01 01:30:00   4656.0
4  2014-07-01 02:00:00   3820.0
```

We have two columns, a `timestamp` and a `value` representing the number of taxi trips in that 30 minute interval.

To use this data with Prophet, we need to rename the columns to `ds` (datestamp) and `y` (value). We‘ll also convert the timestamps to datetime format.

```
df = df.rename(columns={‘timestamp‘: ‘ds‘, ‘value‘: ‘y‘})
df[‘ds‘] = pd.to_datetime(df[‘ds‘])
```

We can plot this data to visualize the time series:

```
ax = df.set_index(‘ds‘).plot(figsize=(12, 8))
ax.set_ylabel(‘Taxi Trips‘)
ax.set_title(‘NYC Taxi Data‘);
```

![NYC taxi data plot](https://33rdsquare.com/taxi_data_plot.png)

From the plot we can see clear seasonal patterns in the data – a repeating daily cycle with lower ridership late at night and spikes during typical commute times. Let‘s see if Prophet can model this and find any anomalies.

### Fitting the Prophet Model

Fitting a Prophet model to the taxi data is straightforward:

```
m = Prophet()
m.fit(df)
```

This fits the model on the entire dataset. We can plot the model components to visualize the break down of the time series into trend, yearly seasonality, and weekly seasonality:

```
m.plot_components(m.predict(df));
```

![Prophet components plot](https://33rdsquare.com/prophet_components.png)

The first plot shows the overall trend – taxi ridership increasing then decreasing over this period. The second plot shows the weekly seasonality (lower on weekends) and the third shows the daily cycle.

### Detecting and Visualizing Anomalies

Now that we have a model of the "normal" behavior of the time series, we can use it to detect outliers that don‘t match the expected patterns.

Prophet makes this easy by automatically identifying outliers when we make predictions with the model. Any large deviations from the model forecast are flagged as anomalies.

To get the forecasted values and anomalies, we can use `prophet.predict`:

```
forecast = m.predict(df)
forecast[‘anomaly‘] = forecast[‘y‘] - forecast[‘yhat‘] > 1.5 * forecast[‘yhat_upper‘]
```

Here we calculate an `anomaly` column that flags any points where the actual `y` value exceeds the upper bound of 95% confidence interval (`yhat_upper`) by more than 50%.

We can then plot the results to visualize the anomalies:

```
fig = m.plot(forecast)
ax = fig.gca()
ax.scatter(df[‘ds‘], df[‘y‘], color=‘r‘, s=4)
ax.scatter(forecast[‘ds‘][forecast[‘anomaly‘]], forecast[‘yhat‘][forecast[‘anomaly‘]], color=‘r‘, s=4)
ax.set_xlabel(‘Date‘)
ax.set_ylabel(‘Taxi Trips‘)
ax.set_title(‘Taxi Data with Anomalies‘)
plt.show()
```

![Anomaly plot](https://33rdsquare.com/taxi_anomalies.png)

The black dots are the actual data points, red dots indicate anomalies, and the blue shaded area is the prediction interval from Prophet.

We can see Prophet flagging several large spikes as anomalies, like the one on New Year‘s Eve. It also catches some smaller anomalies in the troughs that deviate from the normal daily pattern.

## Comparing to Other Methods

Prophet makes time series anomaly detection about as straightforward as it can get, but it‘s not the only option. Some other common approaches include:

- **Statistical methods** like assuming a Gaussian distribution and flagging data points that are >2 or 3 standard deviations from the mean. This can work for finding global outliers but doesn‘t account for seasonality.
- **ARIMA** (Autoregressive Integrated Moving Average) models are a classical time series approach. Points with large residuals from an ARIMA model could indicate anomalies.
- **Machine Learning** approaches like training a supervised classification model to recognize anomalies based on manually labeled examples. This requires having labeled anomaly data upfront.
- **Unsupervised** methods like using clustering, OneClassSVM, or autoencoders to identify data points that differ from the norm without any labels.

Compared to these alternatives, the advantages of Prophet for anomaly detection are:

- Requires few manually set parameters, just feed it a time series
- Models seasonality at different granularities out of the box
- Robustness to missing data and outliers
- Built in support for anomaly detection
- Scalable and production-ready, developed by Facebook

## Deploying to Production

We walked through a simple example here, but how would you actually put automated anomaly detection with Prophet into production to monitor a live stream of data?

The key steps would be:

1. Automatically retrain the Prophet model on a regular basis (e.g. daily) as new time series data comes in
2. For each new data point, use the Prophet predict method to generate a forecast
3. Compare the actual value to the forecasted distribution and flag an anomaly if it exceeds some threshold (e.g. 95% confidence interval)
4. Store the anomalies in a database and/or trigger alerts to engineering teams to investigate
5. Incorporate a mechanism for engineers to provide feedback on whether flagged anomalies were relevant or not, and use this to tune the thresholds

There are a number of platforms, tools, and libraries designed to help streamline this type of workflow, such as the Anomaly Detection API on GCP, Amazon Lookout for Metrics, or open source libraries like Alibi Detect.

But even a simple custom pipeline built around Prophet to model the "normal" behavior and flag deviations can go a long way to automatically catching data quality issues, system failures, or unexpected changes to key business metrics.

## Conclusion

We covered a lot of ground in this post, including:

- What time series anomalies are and why detecting them is important
- An overview of Facebook Prophet and how it models seasonal time series data
- A detailed example of using Prophet to detect anomalies in taxi ridership data
- Comparison to alternative anomaly detection methods
- Recommendations for productionalizing anomaly detection

Hopefully this gives you a solid foundation for tackling anomaly detection on your own time series data. The ease of use and robustness of Prophet make it a great place to start, but the same concepts apply to any method you choose.

Being able to automatically surface unusual data points and use them to improve data quality, identify system issues, and discover business insights is an invaluable skill for any data practitioner. Give Prophet a try and see what anomalies you can find hiding in your metrics.

---

Source: [Detecting Anomalies in Time Series Data with Facebook Prophet](https://33rdsquare.com/anomaly-detection-model-using-facebook-prophet/)