Beginner‘s Guide to Anomaly Detection Using Self-Organizing Maps
Introduction
As a data scientist, one of the most interesting and challenging problems you may come across is anomaly detection – finding patterns in data that deviate significantly from the norm. While there are many techniques for anomaly detection, one particularly powerful unsupervised learning method is the self-organizing map (SOM).
SOMs, also known as Kohonen maps, were invented by Finnish professor Teuvo Kohonen in the 1980s. They are a special class of artificial neural networks that can map high-dimensional input data to a lower-dimensional (usually 2D) grid of neurons, while preserving the topological relationships between the input data points.
This makes SOMs very useful for visualizing and exploring complex datasets to uncover hidden patterns and outliers. Some common applications of SOMs include:
- Detecting credit card fraud from transaction data
- Identifying manufacturing defects from sensor readings
- Finding intrusions in computer networks
- Categorizing documents based on content
The great thing about SOMs is that they learn in an unsupervised manner, without the need for labeled data. This means you can throw in a dataset and let the SOM train itself to identify patterns and anomalies.
In this article, we‘ll dive deep into the world of self-organizing maps. You‘ll learn about the architecture and training process of SOMs, and see how to implement them in Python to detect anomalies. By the end, you‘ll have a solid intuition for this powerful technique that you can apply to uncover insights from your own data. Let‘s get started!
Architecture of Self-Organizing Maps
To understand how self-organizing maps work, let‘s first take a look at their architecture. An SOM consists of a 2D grid of neurons, each of which is fully connected to the input layer.
Each neuron maintains a weight vector of the same dimension as the input vectors. That is, if the input data has 10 features, each neuron will also have a 10-element weight vector. Importantly, there are no lateral connections between the output neurons in an SOM. This is different from other neural network architectures you may be familiar with.
The goal of the SOM is to move the weight vectors of the neurons to match the distribution of the input data, so that each neuron can act as a prototype or representative of a group of similar inputs. Neurons that end up being physically adjacent in the output grid will respond to inputs that are similar to each other. This is how SOMs convert complex high-dimensional data into a simpler 2D representation.
SOM Training Process
Now that you have an idea of what an SOM architecture looks like, let‘s walk through the training process that allows the network to self-organize and learn the patterns in the input data.
The SOM training algorithm can be summarized in the following steps:
- Randomly initialize the weight vectors of all neurons in the grid
- Choose a data point at random from the input dataset
- Find the neuron whose weight vector is closest to the input (this is the best matching unit or BMU)
- Update the weight vector of the BMU to move it closer to the input vector
- Also update the weight vectors of the neurons in the neighborhood of the BMU
- Repeat steps 2-5 for a large number of iterations, gradually decreasing the learning rate and neighborhood size
Let‘s go through each of these steps in more detail. In step 1, the weight vectors of the neurons are initialized to small random values. Then in step 2, an input vector x is selected at random from the training dataset.
In step 3, the BMU is identified by calculating the Euclidean distance between the input and each neuron‘s weight vector. The neuron with the minimum distance is designated as the BMU. You can think of the BMU as the neuron that is most "excited" or responsive to the current input.
Step 4 is where the actual learning happens. The weight vector of the BMU is adjusted to move closer to the input vector. The amount of adjustment depends on a learning rate parameter that decreases over time. This allows the SOM to make large adjustments early in the training process and then fine-tune the weights as it sees more data.
But the BMU isn‘t the only neuron that gets updated. In step 5, the weight vectors of the neurons in the neighborhood of the BMU are also moved closer to the input, with the magnitude of adjustment decaying with distance from the BMU. This is how the SOM preserves the topology of the input space – neurons that are physically adjacent will receive similar updates and thus respond to similar inputs.
Over thousands of iterations, this training process causes the SOM‘s neurons to spread out and cover the input space while retaining their spatial relationships. Regions of high data density will be represented by larger clusters of neurons, while sparse regions will have fewer neurons mapping to them.
Understanding Best Matching Units
As you saw in the training process, the BMU plays a central role in how an SOM learns. At each training step, the BMU is the neuron whose weight vector is closest to the current input. It receives the biggest update and also influences the weights of its neighboring neurons.
The BMU essentially represents the prototype or centroid of a cluster of similar input vectors. As the SOM is trained, each BMU will move to the center of a group of input vectors, while still maintaining its relationship to other neurons.
An important aspect of the SOM training process is that the size of the BMU‘s neighborhood shrinks over time. In the beginning, a large number of neurons around the BMU are updated, allowing the map to quickly spread out and cover the input space. But as training progresses, the neighborhood size decreases, so that only the BMU and its closest neighbors are affected. This allows the map to stabilize and converge to a final configuration.
Comparing SOMs and K-Means Clustering
If you‘re familiar with unsupervised learning techniques, you may notice some similarities between self-organizing maps and k-means clustering. Both methods aim to find a set of prototypes or cluster centers that can represent the input data.
In k-means, you start by specifying the number of clusters you want to find. The algorithm then iteratively assigns each data point to the closest cluster center and updates the centers to be the mean of their assigned points. This process continues until the cluster assignments no longer change.
SOMs are similar in that each neuron acts like a cluster center, and input vectors are mapped to the neuron that is most similar (the BMU). However, there are a few key differences:
-
SOMs impose a 2D topological structure on the cluster centers, while k-means does not.
-
In k-means, only the cluster centers are updated at each iteration. In an SOM, the BMU and its neighbors are all updated together.
-
K-means requires specifying the number of clusters up front, while the number of neurons in an SOM is a hyperparameter that can be tuned.
So while both methods can be used for clustering and dimensionality reduction, SOMs provide a more structured and interpretable output that preserves the spatial relationships between clusters.
Implementing SOMs in Python
Now that you have a solid grasp of the theory behind self-organizing maps, let‘s see how to implement them in Python for an anomaly detection task. We‘ll use a credit card fraud dataset to build an SOM that can identify transactions that are outliers and potentially fraudulent.
First, we‘ll load the necessary libraries and the dataset:
import numpy as np
import pandas as pd
from minisom import MiniSom
from sklearn.preprocessing import MinMaxScaler
data = pd.read_csv(‘credit_card_data.csv‘)
X = data.drop([‘Class‘], axis=1)
y = data[‘Class‘]We‘ll use the minisom library to build our SOM. The input data needs to be scaled to the range [0,1] before training, so we‘ll use scikit-learn‘s MinMaxScaler for that.
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)Now we can create our SOM and specify the hyperparameters:
som_shape = (25,25)
som = MiniSom(x=som_shape[0], y=som_shape[1], input_len=X_scaled.shape[1],
sigma=1.0, learning_rate=0.5)
som.random_weights_init(X_scaled)
som.train_random(data=X_scaled, num_iteration=100)Here we‘re creating a 25×25 SOM with a sigma (neighborhood radius) of 1 and a learning rate of 0.5. We initialize the neuron weights randomly from the input data and train for 100 iterations.
After training, we can visualize the SOM to see how it has mapped the input data:
import matplotlib.pyplot as plt
plt.figure(figsize=(10,10))
frequencies = som.activation_response(X_scaled)
plt.pcolor(frequencies.T, cmap=‘Blues‘)
plt.colorbar()
plt.show()
This plot shows the activation frequencies of each neuron, i.e., how often each neuron was the BMU for the input data. Darker colors indicate higher frequencies. We can see that some neurons are much more active than others, indicating that they have captured clusters of similar data points.
To see which transactions the SOM has flagged as anomalies, we can map the input data to their BMUs and then find the BMUs with the lowest frequencies:
bmu_indexes = som.win_map(X_scaled)
outlier_scores = [1/frequencies[bmu[0], bmu[1]] for bmu in bmu_indexes]
outlier_scores = np.array(outlier_scores)
threshold = np.percentile(outlier_scores, 99)
outliers = X[outlier_scores >= threshold]
print(f"Found {len(outliers)} outliers.")Here we‘re using the inverse of the BMU frequencies as an outlier score, so transactions mapped to less active neurons are more likely to be anomalies. By setting a threshold based on the 99th percentile of the scores, we can identify the top 1% most anomalous transactions.
And that‘s it! With just a few lines of code, we‘ve built an SOM that can detect anomalies in a high-dimensional dataset. Of course, there are many ways to further tune and optimize the model, but this should give you a good starting point for applying SOMs to your own anomaly detection problems.
Conclusion
In this article, we‘ve taken a deep dive into self-organizing maps and how they can be used for anomaly detection. We‘ve explored the architecture and training process of SOMs, understanding how they learn to map high-dimensional data to a 2D grid while preserving topological relationships.
We‘ve also compared SOMs to other clustering methods like k-means, and walked through a practical example of building an SOM in Python for detecting credit card fraud.
Some key takeaways:
- SOMs are a powerful tool for unsupervised learning and anomaly detection, particularly for high-dimensional data
- The training process involves repeatedly finding the BMU for each input and updating its weights and neighborhood
- SOMs provide a more interpretable and structured output than techniques like k-means
- Implementing SOMs in Python is straightforward with libraries like minisom
I hope this article has given you a solid foundation in self-organizing maps and the confidence to start applying them to your own data. Don‘t be afraid to experiment with different architectures, hyperparameters, and visualizations to gain insights from your SOMs.
If you want to learn more, I recommend checking out some of these resources:
- Kohonen‘s original paper on SOMs: https://ieeexplore.ieee.org/document/58325
- A great tutorial on SOMs in Python: https://rubikscode.net/2018/08/20/introduction-to-self-organizing-maps-som/
- The minisom documentation: https://github.com/JustGlowing/minisom
Happy mapping!
