Measuring Semantic Textual Similarity Using BERT Embeddings in Python

Determining how semantically similar two pieces of text are to each other is a fundamental problem in natural language processing (NLP) with a wide range of applications. Use cases for semantic textual similarity include information retrieval, semantic search, duplicate detection, clustering related documents, identifying paraphrases, and more.

At the core of measuring text similarity is the challenge of how to best represent text in a mathematical form that captures its underlying meaning. With the rise of powerful neural language models like BERT that can generate rich contextual word embeddings, we now have an effective tool for encoding the semantics of text as dense numeric vectors.

In this article, we‘ll dive into using BERT to quantify the semantic similarity between text snippets in Python. We‘ll cover the core concepts, walk through the implementation process step-by-step, and explore how to get the most out of this approach. Basic familiarity with NLP and Python will be helpful, but we‘ll explain the key ideas along the way.

From Text to Vectors: Semantic Representation

The traditional approach in NLP was to represent text as a bag-of-words – a sparse vector capturing the frequency of words but ignoring order and context. While simple, this fails to capture the rich relational structure and semantics of language.

Word embeddings, as popularized by methods like word2vec and GloVe, brought a new paradigm of mapping words to dense vectors that locate semantically similar words close together in the embedding space. By using the word embeddings of the constituent words, we can construct an embedding representation for a full sentence or paragraph.

However, standard word embeddings have limitations – each word receives a single fixed vector representation regardless of the context. The meaning and usage of words can vary significantly based on their context, so capturing this is crucial for more nuanced representation.

BERT: Bidirectional Contextual Word Embeddings

BERT (Bidirectional Encoder Representations from Transformers) is a breakthrough NLP model developed by Google that addresses the shortcomings of context-free embeddings. BERT uses the Transformer architecture to generate contextual word embeddings – vector representations that dynamically adapt based on the surrounding words.

BERT is pre-trained in a self-supervised way on a massive amount of unlabeled text, using two clever training objectives that enable learning from plain text alone:

1. Masked Language Modeling – some percentage of words are masked out, and the model learns to predict the masked words based on the unmasked context
2. Next Sentence Prediction – the model is fed pairs of sentences and learns to predict if the second sentence actually follows the first or is just a random distractor

Through this pre-training process, BERT learns a powerful general-purpose representation of language that can transfer to a variety of downstream NLP tasks. We can then fine-tune BERT or extract its embeddings for use in our specific applications.

Generating BERT Embeddings

To utilize BERT for measuring semantic similarity, we first need to generate the contextual word embeddings. We‘ll use the transformers library by Hugging Face which provides a simple API for working with pre-trained BERT models.

First install the library:

pip install transformers

Then we can load a pre-trained BERT model and its associated tokenizer:

from transformers import BertModel, BertTokenizer

model_name = ‘bert-base-uncased‘ 
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertModel.from_pretrained(model_name)

Here we‘re using the bert-base-uncased version which is a smaller model trained on lowercase English text. There are larger, cased, and multilingual BERT models available that can give better performance at the cost of increased compute requirements.

Next, let‘s define a function to generate BERT embeddings for a given text input:

import torch

def get_bert_embedding(text, model, tokenizer):
    encoded_input = tokenizer(text, return_tensors=‘pt‘)
    with torch.no_grad():
        output = model(**encoded_input)

    last_hidden_states = output.last_hidden_state
    return last_hidden_states[0].mean(dim=0)

This function handles the following:

1. Tokenizes the input text into a sequence of tokens BERT understands
2. Feeds the tokens into the BERT model to generate the contextual embeddings
3. Extracts the embeddings from BERT‘s last hidden layer
4. Takes the mean of the embeddings along the sequence length dimension to get a sentence-level representation

The result is a single 768-dimensional vector encoding the semantic content of the full input text.

Computing Semantic Similarity

With the BERT embedding function in hand, measuring the semantic similarity between a pair of text snippets is straightforward:

from scipy.spatial.distance import cosine

text1 = "Apple‘s revenue jumped 11% to a record high last quarter"
text2 = "Apple posted strong third quarter financial results beating analyst expectations"

embedding1 = get_bert_embedding(text1, model, tokenizer)
embedding2 = get_bert_embedding(text2, model, tokenizer)

similarity = 1 - cosine(embedding1, embedding2)
print(f‘Semantic similarity: {similarity:.3f}‘)

Output:

Semantic similarity: 0.852

Here we simply take the cosine similarity between the BERT embeddings of the two texts. The cosine similarity is the cosine of the angle between the vectors, ranging from 0 for orthogonal vectors to 1 for identical vectors. Subtracting from 1 gives us a similarity score where higher values indicate more similar meanings.

We can see BERT correctly recognizes the two sentences discuss very similar information about Apple‘s strong financial performance, despite using quite different wording.

For comparison, let‘s look at a negative example:

text1 = "Apple‘s revenue jumped 11% to a record high last quarter"
text2 = "Orange juice futures dropped on weakening demand from China"

embedding1 = get_bert_embedding(text1, model, tokenizer)
embedding2 = get_bert_embedding(text2, model, tokenizer)

similarity = 1 - cosine(embedding1, embedding2)  
print(f‘Semantic similarity: {similarity:.3f}‘)

Output:

Semantic similarity: 0.212

As expected, BERT assigns a much lower similarity score to these unrelated sentences. The presence of related terms like "Apple" and "Orange" isn‘t enough to fool the model which understands the sentences are talking about very different topics.

Similarity At Scale with sentence-transformers

For most real-world applications, we‘ll want to compute semantic similarity between many pairs or larger sets of text snippets. The sentence-transformers library extends the Hugging Face ecosystem with added functionality for semantic similarity and semantic search.

Install it with:

pip install -U sentence-transformers

And we can now easily encode a corpus of text snippets into embeddings:

from sentence_transformers import SentenceTransformer

sentences = [
    "Apple reports strong third quarter results",
    "Google announces a new version of its popular search engine",  
    "Apple‘s phone sales increase by 10% year-over-year",
    "Amazon invests in new warehouses to meet growing demand",
    "Google introduces new AI-powered features for Gmail"
]

model = SentenceTransformer(‘all-mpnet-base-v2‘)
embeddings = model.encode(sentences)

Here we‘re using the all-mpnet-base-v2 model which is an improved model that tends to perform better than standard BERT models for sentence-level semantic similarity.

The result is a 2D matrix of embeddings we can use to rapidly compute the pairwise similarities between all sentences:

from sklearn.metrics.pairwise import cosine_similarity

similarities = cosine_similarity(embeddings)
print(similarities)

Output:

[[1.         0.18468076 0.75101781 0.22651745 0.25964141]
 [0.18468076 1.         0.26708157 0.1487712  0.50037128]
 [0.75101781 0.26708157 1.         0.23701436 0.29559305]
 [0.22651745 0.1487712  0.23701436 1.         0.15851179]
 [0.25964141 0.50037128 0.29559305 0.15851179 1.        ]]

We can see the model correctly groups the Apple-related sentences and Google-related sentences as more similar to each other while considering the Amazon sentence to be unrelated to the others.

Limitations and Extensions

While BERT embeddings are a powerful tool for textual similarity, there are some key limitations to be aware of:

• Relying on simple mean pooling of word embeddings to represent sentences is convenient but throws away useful information. More advanced pooling strategies like using the [CLS] token embedding, max pooling, or weighted averaging can improve results.

• Sentences longer than BERT‘s max input length (512 tokens) will be truncated, so information at the end of very long texts won‘t be captured. Dealing with longer texts may require splitting into chunks.

• Fine-tuning BERT on your specific domain or task, rather than relying on the default pre-training, can yield significant performance gains if you have sufficient labeled data.

• Running BERT can be computationally intensive, especially for long texts or large-scale processing. Distilled versions like DistilBERT can be good options for faster inference.

• BERT‘s underlying Transformer architecture has been iterated on and newer models like RoBERTa, XLNet, ELECTRA etc. can outperform it in many scenarios. The same general approach applies but it‘s good to explore different models.

Despite these caveats, semantic similarity with BERT embeddings remains a go-to approach that delivers strong results across a range of practical applications. With active developments in Transformer-based language models, its effectiveness will only keep growing.

Conclusion

Semantic textual similarity is a core NLP task that BERT has revolutionized with its powerful contextual embeddings. As we‘ve seen, modern Python libraries make it quite straightforward to harness BERT for applications like semantic search, clustering, duplicate detection, and more.

While challenges remain in effectively representing longer texts and more nuanced aspects of language, the basic recipe of getting high-quality dense vector embeddings and measuring their similarity is a proven approach. Continued advances in language models and deep metric learning hold great promise for further pushing the boundaries.

Hopefully this guide has equipped you with a solid practical understanding of semantic textual similarity with BERT and how to implement it in Python. Try applying these techniques to your own datasets and let us know how it goes!