Detecting Anomalies in Multivariate Time Series Data Using Vector Autoregression (VAR)

Anomaly detection is a critical task in many domains, from fraud detection in financial transactions to identifying malfunctioning sensors in industrial equipment. The goal is to automatically identify unusual patterns or instances that deviate significantly from the norm. While many anomaly detection algorithms exist for tabular data, detecting anomalies in time series data requires special considerations due to its sequential nature and potential for temporal dependencies.

In this article, we‘ll dive into anomaly detection for multivariate time series data using a powerful technique called Vector Autoregression (VAR). We‘ll explain the concepts step-by-step and demonstrate how to implement this approach using Python. Whether you‘re a data scientist, analyst, or researcher, understanding how to spot anomalies in your time series data is a valuable skill to have in your toolbox.

Why Anomaly Detection for Time Series is Different

Time series data, where observations are recorded chronologically, differs from regular cross-sectional data in several key ways:

1. Sequential dependency: Observations in a time series are typically dependent on past values. This temporal relationship must be accounted for in modeling.

2. Autocorrelation: Time series often exhibit similarity between observations as a function of the time lag between them. Removing autocorrelation through techniques like differencing is often necessary.

3. Stationarity: Many time series models assume the data is stationary, meaning its statistical properties like mean and variance are constant over time. Non-stationary data needs to be transformed before modeling.

4. Seasonality and trends: Time series can have recurring patterns or long-term trends that should be addressed separately from anomaly detection.

Because of these characteristics, using standard anomaly detection algorithms on time series data without proper preprocessing and modeling can lead to spurious results. Instead, we need to use techniques specifically designed for handling sequential data.

Univariate vs. Multivariate Time Series

Another important distinction is between univariate and multivariate time series. A univariate time series consists of a single variable tracked over time, such as daily stock prices or monthly sales figures. In contrast, multivariate time series involve multiple interconnected variables observed simultaneously over time.

Examples of multivariate time series include:

• Vital signs like heart rate, blood pressure, and oxygen levels measured continuously for a patient
• Temperature, humidity, and wind speed readings from a weather station
• Website metrics like page views, unique visitors, and bounce rate recorded hourly

Univariate time series can be modeled using techniques like Autoregression (AR), Moving Average (MA), and combinations thereof (ARMA, ARIMA). However, for multivariate time series, we need models that can capture the dynamic relationships between multiple variables over time, which is where Vector Autoregression (VAR) comes in.

A Primer on Vector Autoregression (VAR)

Vector Autoregression (VAR) is a stochastic process model used to capture the linear interdependencies among multiple time series. It is one of the most widely used models for analyzing multivariate time series data.

In a VAR model, each variable is modeled as a linear function of its own past values, the past values of the other variables in the system, and an error term. Mathematically, a VAR(p) model with n variables can be written as:

y(t) = c + A(1)y(t-1) + … + A(p)y(t-p) + e(t)

where:

• y(t) is an n × 1 vector of the n variables at time t
• c is an n × 1 vector of constants
• A(i) are n × n coefficient matrices for each lag i = 1, …, p
• e(t) is an n × 1 vector of white noise error terms

The lag order p specifies how many previous time steps are used as predictors. Choosing the optimal p is an important step in building a VAR model, which we‘ll discuss later.

Some key assumptions of the VAR model are:

1. Stationarity: The time series should be stationary, either inherently or after differencing.
2. Linearity: The relationships between variables are assumed to be linear.
3. Homoscedasticity: The variance of the error terms is assumed to be constant over time.
4. No serial correlation: The error terms should not be correlated with their own lagged values.

If these assumptions are violated, extensions of the basic VAR model, such as the Vector Error Correction Model (VECM) or structural VAR models, may be more appropriate.

Preparing Multivariate Time Series Data for Modeling

Before we can build a VAR model for anomaly detection, we need to preprocess our multivariate time series data. The most important step is checking for stationarity and converting non-stationary series to stationary ones.

A stationary time series has constant mean, variance, and autocovariance over time. Non-stationary behavior, such as trends or seasonality, can obscure the underlying dynamics and lead to spurious results in modeling. We can check for stationarity using statistical tests like the Augmented Dickey-Fuller (ADF) test or the Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test.

If a series is found to be non-stationary, we can apply transformations like differencing, detrending, or seasonal adjustment to make it stationary. Differencing involves computing the differences between consecutive observations, which can remove trends. Seasonal differencing extends this idea to remove seasonal patterns.

Here‘s an example of checking for stationarity using the ADF test in Python:

from statsmodels.tsa.stattools import adfuller

def adf_test(series, signif=0.05):
    result = adfuller(series)
    print(f‘ADF Statistic: {result[0]}‘)
    print(f‘p-value: {result[1]}‘)
    if result[1] <= signif:
        print(‘Reject null hypothesis - Data is stationary‘)
    else:
        print(‘Fail to reject null hypothesis - Data is non-stationary‘)

# Example usage        
adf_test(data[‘variable1‘])

If the p-value is less than the chosen significance level (e.g., 0.05), we reject the null hypothesis that the series is non-stationary.

Once we have stationary series for all variables, we can proceed to fitting the VAR model. It‘s also a good idea to standardize the data to have zero mean and unit variance, which can improve the numerical stability of the model fitting process.

Building a VAR Model for Anomaly Detection

With our preprocessed data in hand, we‘re ready to build a VAR model for anomaly detection. The general steps are:

1. Select the optimal lag order (p) for the model
2. Fit the VAR model to the training data
3. Compute the residuals (forecast errors) for each time step
4. Set a threshold for anomaly scores based on the residuals
5. Flag time points whose anomaly scores exceed the threshold

Let‘s go through each step in more detail.

Selecting the Lag Order

The lag order p determines how many previous time steps are used to predict the current values. A larger p allows capturing longer-term dependencies but also increases model complexity and computational cost.

We can use information criteria like the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) to select the optimal lag order. These metrics balance model fit and parsimony. We choose the lag order that minimizes the criterion.

In Python, we can use the VAR class from the statsmodels package to select the lag order:

from statsmodels.tsa.api import VAR

var_model = VAR(data)
lag_order = var_model.select_order(maxlags=10) 
print(f‘Optimal lag order: {lag_order.aic}‘)

This code fits VAR models with lag orders up to 10 and selects the one with the minimum AIC.

Fitting the VAR Model

Once we‘ve selected the lag order, we can fit the VAR model to our training data. This estimates the coefficient matrices A(i) and the covariance matrix of the error terms.

var_results = var_model.fit(lag_order.aic)
print(var_results.summary())

The summary() method provides diagnostic information about the fitted model, such as the estimated coefficients, standard errors, and goodness-of-fit measures.

Computing Residuals and Anomaly Scores

The key idea behind using a VAR model for anomaly detection is that anomalies will have large residuals (differences between actual and predicted values) compared to normal instances. We can compute the residuals for each time step and use them as anomaly scores.

residuals = var_results.resid
anomaly_scores = np.sum(residuals**2, axis=1)

Here, we square the residuals and sum across variables to get a scalar anomaly score for each time point.

Setting an Anomaly Threshold

To flag anomalies, we need to set a threshold above which an anomaly score is considered unusual. There are various ways to do this, such as:

• A simple standard deviation-based threshold, e.g., the mean anomaly score plus 2-3 standard deviations
• A quantile-based threshold, e.g., the 95th or 99th percentile of anomaly scores
• An empirical threshold based on domain knowledge or labeled anomalies

threshold = np.mean(anomaly_scores) + 3*np.std(anomaly_scores)
is_anomaly = anomaly_scores > threshold
print(f‘Number of anomalies detected: {np.sum(is_anomaly)}‘)

We can then visualize the anomalies on a plot of the original time series:

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(10, 6))
ax.plot(data[‘variable1‘], label=‘Variable 1‘)
ax.scatter(data.index[is_anomaly], data[‘variable1‘][is_anomaly], color=‘red‘, label=‘Anomalies‘)
ax.legend()
plt.show()

This will highlight the anomalous time points in red on the plot.

Comparing VAR to Other Multivariate Time Series Models

While VAR is a popular choice for modeling multivariate time series, there are other techniques that may be useful depending on the characteristics of your data and the goals of your analysis. Some related models include:

• Vector Moving Average (VMA): A model where each variable depends on the past errors of all variables in the system, rather than their past values as in VAR.

• Vector Autoregressive Moving Average (VARMA): A generalization of VAR and VMA that includes both lagged values and errors.

• Vector Error Correction Model (VECM): An extension of VAR that accounts for cointegration, i.e., long-run equilibrium relationships between non-stationary variables.

• Structural VAR (SVAR): A variant of VAR that incorporates economic theory or expert knowledge to impose constraints on the model parameters.

• Dynamic Factor Models (DFM): A dimensionality reduction technique that explains the co-movements of multiple time series using a smaller number of latent factors.

The choice of model depends on factors such as the presence of moving average terms, cointegration, interpretability requirements, and computational feasibility.

Best Practices and Considerations

When using VAR for anomaly detection in practice, there are several things to keep in mind:

1. Data preprocessing: Ensure your time series are properly cleaned, aligned, and formatted before modeling. Handle missing values and outliers appropriately.

2. Stationarity: Carefully check for stationarity and apply necessary transformations. Non-stationary data can lead to spurious results.

3. Training-test split: Use a rolling window or expanding window approach to split your data into training and testing sets over time. Avoid random splits that leak future information.

4. Model diagnostics: Check the assumptions of your VAR model, such as residual normality and homoscedasticity. Use residual plots and tests to assess model fit.

5. Hyperparameter tuning: Experiment with different lag orders and regularization techniques to optimize model performance and avoid overfitting.

6. Anomaly validation: When possible, validate detected anomalies with domain experts or additional data sources to assess their relevance and impact.

7. Scalability: VAR models can become computationally intensive for high-dimensional data. Consider dimensionality reduction techniques or parallel computing for large-scale applications.

Conclusion

Anomaly detection in multivariate time series data is a challenging but important task in many domains. By leveraging the power of Vector Autoregression (VAR) models, we can capture the complex temporal dependencies between multiple variables and identify unusual patterns that deviate from normal behavior.

In this article, we‘ve walked through the process of building a VAR-based anomaly detection system step-by-step, from data preprocessing to model selection, fitting, and evaluation. We‘ve discussed best practices and considerations for applying this approach in practice, as well as alternative models for different scenarios.

Equipped with this knowledge, you‘re now ready to tackle anomaly detection in your own multivariate time series data. Remember to carefully preprocess your data, select appropriate model parameters, and validate your results. With the right tools and techniques, you can uncover valuable insights and drive better decision-making from your time series data.