A Beginner‘s Guide to Image Similarity Using Python

Introduction

Image similarity is a fundamental problem in computer vision and multimedia retrieval. Whether you want to organize your personal photo collection, build a visual search engine, or implement a content moderation system, you need a way to quantify how alike two images are.

This used to require complex, hand-crafted algorithms – but recent breakthroughs in deep learning have made powerful image similarity models accessible to everyone. In this beginner‘s guide, we‘ll explore both traditional and modern approaches, and walk through building your own image similarity system using Python.

What is Image Similarity?

At its core, image similarity is about representing images numerically in a way that captures their visual content, so that they can be compared mathematically. There are two main approaches:

1. Traditional Methods: These extract hand-engineered features like color histograms, textures, and shapes. Images are compared based on distance metrics between these feature vectors.

2. Deep Learning Methods: These learn image representations directly from data using convolutional neural networks (CNNs). Images are mapped to embedding vectors which can be compared using distance or similarity metrics.

Deep learning has largely surpassed traditional methods in terms of accuracy and robustness. For example, a 2020 paper benchmarking image similarity techniques found that:

Method	Accuracy
Color Histogram	41.2%
HOG	56.5%
VGG-16 CNN	83.7%

However, traditional methods can still be useful for quick prototyping and understanding core concepts. So let‘s start there and build up to the state-of-the-art!

Building a Simple Image Similarity System

We‘ll start by implementing a basic image similarity system using color histograms and Euclidean distance. Here‘s the step-by-step process:

1. Read images and convert to a common color space
2. Compute color histograms
3. Flatten histograms into feature vectors
4. Define a distance function
5. Compare a query image to a database of reference images
6. Retrieve the most similar reference images

Step 1: Read Images

First we need to load our image data. We‘ll use the Pillow library to read images and convert them to the HSV color space, which tends to work better for similarity than RGB.

from PIL import Image

def load_image(path):
    img = Image.open(path)
    return img.convert(‘HSV‘)  

ref_img1 = load_image("ref1.jpg")
ref_img2 = load_image("ref2.jpg")  
query_img = load_image("query.jpg")

Step 2: Compute Color Histograms

Next we‘ll compute a color histogram for each image. This counts the frequency of each color value, quantizing the full space to a fixed number of bins. We‘ll use NumPy to easily compute histograms across all three HSV channels.

import numpy as np

def color_histogram(img, bins=(8,8,8)):
    hist = np.histogram(img, bins=bins, range=((0,180),(0,256),(0,256)))
    return hist[0]  

ref_hist1 = color_histogram(ref_img1)
ref_hist2 = color_histogram(ref_img2)
query_hist = color_histogram(query_img)  

Step 3: Flatten Histograms

To compare histograms, we need to flatten them into 1D feature vectors.

ref_feat1 = ref_hist1.flatten()
ref_feat2 = ref_hist2.flatten()
query_feat = query_hist.flatten()

Step 4: Define Distance Function

We‘ll use Euclidean distance to compare histogram feature vectors. SciPy provides an optimized implementation.

from scipy.spatial.distance import euclidean

def image_distance(feat1, feat2):
    return euclidean(feat1, feat2)

Step 5: Compare Query to References

Now we can compare our query image to each reference image by calculating the distance between their feature vectors.

dist1 = image_distance(query_feat, ref_feat1) 
dist2 = image_distance(query_feat, ref_feat2)

print(f"Distance to ref1: {dist1:.2f}")
print(f"Distance to ref2: {dist2:.2f}")

Step 6: Retrieve Most Similar

Finally, we sort the reference images by their distance to the query and return the closest matches.

results = sorted([(dist1, ‘ref1‘), (dist2, ‘ref2‘)])

print(f"Top match: {results[0][1]}")
print(f"2nd best match: {results[1][1]}")

And there we have it – a working image similarity system in just a few lines of code! Try it out on your own images and see what results you get.

Of course, this basic approach has major limitations. Color histograms throw away all spatial information, so images with similar colors but different contents will be retrieved (false positives). Conversely, images of the same scene under different lighting or viewpoints will look dissimilar (false negatives).

Advanced Techniques

To improve on the simple global color histogram, we need both more discriminative image features and more robust similarity metrics. Here are some key ideas:

• Local Features: Instead of a single global histogram, partition the image into regions and describe each one with color, texture, and shape features. Popular local descriptors include SIFT, SURF, and ORB.

• Bag-of-Visual-Words: Treat local features like words and represent images as histograms over a "visual vocabulary". This brings powerful NLP techniques to image analysis.

• Earth Mover‘s Distance: Compare histograms based on the minimal cost to transform one into the other. This is more robust than Euclidean distance to small shifts and mismatches.

However, the biggest leaps in image similarity have come from deep learning. Convolutional neural networks trained on image classification learn rich, semantic feature representations that transfer well to measuring visual similarity. There are a few ways to leverage this:

1. Take a pre-trained CNN and use the features from its last layer before the output as a descriptor. For example, VGG-16 trained on ImageNet classification gives a 4096-dimensional feature.

2. Train a CNN explicitly for similarity using loss functions like contrastive loss or triplet loss. These push matching images to have close descriptors and non-matching images to be far apart.

3. Fine-tune a pre-trained model on a task-specific similarity dataset using metric learning. This adapts the representation to your target domain while benefiting from general features learned at scale.

The CVPR 2020 tutorial on representation learning covers recent advances in self-supervised and unsupervised learning that yield even more powerful and generalizable image features. Vision transformers have also shown promising results by bringing NLP sequence models to image encoding.

Building a CNN Image Similarity Service

For a more practical deep learning example, let‘s walkthrough setting up an image similarity service using a pre-trained CNN and the Flask web framework. We‘ll use the VGG-16 model which you can download from Kaggle.

Step 1: Install Dependencies

First make sure you have the necessary Python packages. You‘ll need Flask for the web service, TensorFlow for the CNN model, and NumPy.

pip install flask tensorflow numpy

Step 2: Load Pre-trained Model

Load the VGG-16 model and remove the final classification layer to expose the inner feature representation. We‘ll use TensorFlow, but the same approach works with PyTorch or Keras.

import tensorflow as tf
from tensorflow.keras.applications import vgg16

model = vgg16.VGG16(weights=‘vgg16_weights.h5‘)
model = tf.keras.Model(inputs=model.input, outputs=model.layers[-2].output)

Step 3: Define Feature Extraction

Write a function to preprocess an input image and pass it through the model to extract a feature vector.

def extract_features(img):
    img = img.resize((224, 224))
    img = vgg16.preprocess_input(np.array(img))
    features = model.predict(img[np.newaxis, ...])
    return features.flatten()

Step 4: Build Flask API

Create a Flask route to handle image similarity queries. It should receive a query image, compare it to a database of pre-computed reference features, and return the most similar matches.

from flask import Flask, request, jsonify

app = Flask(__name__)

# Load reference features 
ref_feats = np.load(‘ref_feats.npy‘)

@app.route(‘/search‘, methods=[‘POST‘])
def image_search():
    file = request.files[‘image‘]
    img = Image.open(file.stream)

    query_feat = extract_features(img)

    dists = np.linalg.norm(ref_feats - query_feat, axis=1)  
    ids = np.argsort(dists)[:5]

    return jsonify({‘results‘: ids.tolist()})

if __name__ == ‘__main__‘:
    app.run()

Step 5: Test the Service

Run the Flask server and send it a query image to test the similarity search.

curl -X POST -F "[email protected]" http://localhost:5000/search

This should return the IDs of the top 5 most similar reference images. Of course, this is just a simple example – a real production service would need to consider factors like:

• Efficiently indexing and searching a large database of reference images
• Caching and load balancing for high query throughput
• Filtering and deduplicating results
• Monitoring and logging

But it demonstrates the core components of an image similarity system using deep learning. You can find complete code examples for this and other approaches in the excellent Image Similarity Experiments repo.

Conclusion

In this guide, we‘ve covered the key concepts and techniques for tackling image similarity, from simple color histograms to state-of-the-art CNN models. Some key takeaways:

• Image similarity is about extracting numeric features that capture relevant visual content and comparing them with distance metrics
• Traditional methods use hand-engineered features while deep learning approaches learn features directly from data
• CNN models pre-trained on image classification provide powerful off-the-shelf feature extractors
• Specialized training with metric learning losses can further improve similarity models
• Self-supervised learning and vision transformers are promising new directions

We walked through code examples of a basic histogram similarity system in pure Python/NumPy, as well as a more sophisticated CNN similarity service using TensorFlow and Flask.

However, we‘ve only scratched the surface of this rich field. Other important topics to explore include:

• Graph-based similarity methods for large-scale retrieval
• Hashing and quantization for compact image representations
• Multimodal similarity across images, text, audio, etc.
• Similarity explanations and visual search interfaces
• Domain-specific applications like medical image retrieval, fashion recommendations, and visual product search

I encourage you to dive deeper into the papers and code examples linked throughout this guide, and to try implementing and extending these techniques yourself. Image similarity is a challenging but rewarding problem that encompasses both fascinating research and impactful real-world applications. Have fun exploring it!