Part 6: Step by Step Guide to Master NLP – Word2Vec

Introduction

Welcome to Part 6 of the Step by Step Guide to Natural Language Processing series! In the previous parts, we covered various foundational concepts in NLP such as text preprocessing, language modeling, and statistical word embeddings. In this post, we will dive deep into one of the most popular and influential word embedding techniques – Word2Vec.

Word embeddings have revolutionized the field of NLP by enabling machine learning models to understand and reason about the meaning and relationships between words. However, traditional word embedding methods like count-based and statistical approaches had several limitations in terms of scalability, efficiency, and capturing semantic relationships.

This is where Word2Vec comes in. Proposed by a team of researchers at Google led by Tomas Mikolov in 2013, Word2Vec is a prediction-based model that can learn high-quality word embeddings from large amounts of unstructured text data in an unsupervised manner. It has become the go-to technique for generating dense vector representations of words that capture both syntactic and semantic similarities.

In this post, we will take a detailed look at the inner workings of Word2Vec and its two main architectures – Continuous Bag-of-Words (CBOW) and Skip-Gram. We will also discuss the advantages and limitations of each architecture and provide guidelines on when to use which one. By the end of this post, you will have a solid understanding of Word2Vec and how it powers various downstream NLP applications. Let‘s get started!

Recap of Word Embeddings

Before we jump into Word2Vec, let‘s quickly recap what word embeddings are and why they are important in NLP.

Word embeddings are a type of word representation that allows words with similar meaning to have similar representations. They are dense, low-dimensional vectors that capture the semantic and syntactic relationships between words in a language.

The main idea behind word embeddings is the distributional hypothesis, which states that words that occur in similar contexts tend to have similar meanings. By learning word embeddings from large volumes of text data, we can map words into a continuous vector space where semantically similar words are closer to each other.

Word embeddings have several advantages over traditional word representations like one-hot encoding:

1. They can capture complex linguistic relationships between words like analogies and similarities.
2. They are dense and low-dimensional, making them computationally efficient.
3. They can be learned automatically from raw text data in an unsupervised manner.
4. They can be used as input features for various downstream NLP tasks like text classification, named entity recognition, machine translation, etc.

However, traditional methods for learning word embeddings like count-based and statistical approaches had several limitations:

1. They were based on the co-occurrence matrix which could be very high-dimensional and sparse.
2. They could not capture the complex non-linear relationships between words.
3. They were computationally expensive and did not scale well to large vocabularies and corpora.
4. They struggled to handle rare and out-of-vocabulary words.

This is where prediction-based models like Word2Vec came into the picture and revolutionized the field of word embeddings.

Introducing Word2Vec

Word2Vec is a group of neural network models that can learn dense word embeddings from large amounts of unstructured text data in an unsupervised fashion. It was introduced by Tomas Mikolov and his colleagues at Google in their seminal paper titled "Efficient Estimation of Word Representations in Vector Space" in 2013.

The main idea behind Word2Vec is to predict the surrounding words given a center word, or vice versa. By training a neural network on this prediction task, Word2Vec can learn meaningful word embeddings that capture the semantic and syntactic relationships between words.

There are two main architectures used in Word2Vec:

1. Continuous Bag-of-Words (CBOW)
2. Skip-Gram

Both architectures have their own strengths and weaknesses, which we will discuss in detail in the following sections. But first, let‘s understand the common building blocks of these architectures.

Building Blocks of Word2Vec

Both CBOW and Skip-Gram architectures in Word2Vec use a simple neural network with a single hidden layer. The input to the network is a one-hot encoded vector representing a word, and the output is a probability distribution over the vocabulary indicating the likelihood of each word being the context word.

The key components of the Word2Vec neural network are:

1. Input layer: A one-hot encoded vector of size V (vocabulary size) representing the input word.
2. Hidden layer: A fully connected layer with N neurons, where N is the desired dimensionality of the word embeddings. This layer maps the input word to a dense, low-dimensional vector.
3. Output layer: A softmax layer with V neurons that outputs the probability distribution over the vocabulary.
4. Weight matrices: There are two weight matrices in the network – the input-to-hidden matrix W of size V x N, and the hidden-to-output matrix W‘ of size N x V.

During training, the network is fed with word pairs (input word, context word) extracted from a large text corpus using a sliding window approach. The objective is to maximize the probability of predicting the context word given the input word (or vice versa in Skip-Gram).

After training, the learned input-to-hidden weight matrix W acts as the word embedding matrix, where each row represents the embedding vector for a word in the vocabulary. These embedding vectors can then be used as input features for various downstream NLP tasks.

Now let‘s dive deeper into the two architectures used in Word2Vec – CBOW and Skip-Gram.

Continuous Bag-of-Words (CBOW)

The Continuous Bag-of-Words (CBOW) architecture in Word2Vec predicts the center word given the surrounding context words. The context words are the words that appear within a fixed-size window around the center word.

Here‘s how the CBOW architecture works:

1. The input to the network is a set of one-hot encoded vectors representing the context words.
2. The one-hot vectors are multiplied with the input-to-hidden weight matrix W to obtain the hidden layer activations. Since the context words are represented as a bag-of-words, the hidden layer activations are averaged to obtain a single vector.
3. The averaged hidden layer vector is then multiplied with the hidden-to-output weight matrix W‘ to obtain the output scores.
4. The output scores are passed through a softmax function to obtain the probability distribution over the vocabulary, indicating the likelihood of each word being the center word.
5. The network is trained using cross-entropy loss to maximize the probability of predicting the correct center word.

The objective function for the CBOW architecture is:

$$J{CBOW}(\theta) = -\frac{1}{T} \sum{t=1}^{T} \log p(wt | w{t-c}, …, w{t-1}, w{t+1}, …, w_{t+c})$$

where $wt$ is the center word, $w{t-c}, …, w{t-1}, w{t+1}, …, w_{t+c}$ are the context words, $c$ is the context window size, and $\theta$ represents the model parameters (weight matrices).

The advantages of the CBOW architecture are:

1. It is faster to train than Skip-Gram since it averages the context words.
2. It works well for frequent words and can capture their semantics accurately.

However, the limitations of CBOW are:

1. It may struggle to capture the semantics of rare words since they get averaged with other context words.
2. It may not be able to differentiate between multiple meanings of a word based on context (e.g. "bank" as a financial institution vs. a river bank).

Skip-Gram

The Skip-Gram architecture in Word2Vec does the opposite of CBOW – it predicts the surrounding context words given the center word. This allows the model to learn more fine-grained representations of words and capture their multiple meanings based on context.

Here‘s how the Skip-Gram architecture works:

1. The input to the network is a one-hot encoded vector representing the center word.
2. The one-hot vector is multiplied with the input-to-hidden weight matrix W to obtain the hidden layer activation.
3. The hidden layer activation is then multiplied with the hidden-to-output weight matrix W‘ to obtain the output scores for each context word.
4. The output scores are passed through a softmax function to obtain the probability distribution over the vocabulary, indicating the likelihood of each word being a context word.
5. The network is trained using cross-entropy loss to maximize the average probability of predicting the correct context words.

The objective function for the Skip-Gram architecture is:

$$J{Skip-Gram}(\theta) = -\frac{1}{T} \sum{t=1}^{T} \sum{-c \leq j \leq c, j \neq 0} \log p(w{t+j} | w_t)$$

where $wt$ is the center word, $w{t+j}$ are the context words, $c$ is the context window size, and $\theta$ represents the model parameters.

The advantages of the Skip-Gram architecture are:

1. It can learn more fine-grained representations of words and capture their multiple meanings based on context.
2. It works well for rare words and can learn their embeddings accurately.

However, the limitations of Skip-Gram are:

1. It is slower to train than CBOW since it predicts each context word separately.
2. It may require more training data to learn high-quality embeddings for frequent words.

Recent Advancements and Variants

Since the introduction of Word2Vec in 2013, there have been several advancements and variants proposed to improve its performance and address its limitations. Some of the notable ones are:

1. Negative Sampling: A more efficient way to train Word2Vec by using only a small number of "negative" samples instead of the entire vocabulary for each update.
2. Subword Embeddings: Techniques like FastText and Byte Pair Encoding (BPE) that learn embeddings for subwords (character n-grams or word pieces) to better handle out-of-vocabulary words and morphological variations.
3. Contextualized Word Embeddings: Models like ELMo and BERT that learn context-dependent embeddings for words based on their surrounding sentences, allowing for more accurate representations.

Project Ideas and Applications

Word2Vec embeddings have been widely used in various NLP applications and have served as a foundation for more advanced techniques. Here are some project ideas and applications where Word2Vec can be useful:

1. Text Classification: Use Word2Vec embeddings as input features for classifying text into predefined categories like sentiment analysis, topic classification, spam detection, etc.
2. Named Entity Recognition: Utilize Word2Vec embeddings to identify and classify named entities like person names, locations, organizations, etc. in text.
3. Machine Translation: Incorporate Word2Vec embeddings into sequence-to-sequence models for machine translation to improve the quality of translations.
4. Text Summarization: Leverage Word2Vec embeddings to identify important sentences and generate coherent summaries of long documents.
5. Recommendation Systems: Use Word2Vec to learn embeddings for items (e.g. products, movies) based on their text descriptions and user interactions, and recommend similar items to users.

Conclusion

In this post, we took a deep dive into Word2Vec, a powerful word embedding technique that has revolutionized the field of NLP. We started with a recap of word embeddings and their importance, and then introduced Word2Vec and its two main architectures – CBOW and Skip-Gram.

We discussed the technical details of each architecture, including their objective functions, advantages, and limitations. We also covered some recent advancements and variants of Word2Vec that address its shortcomings and improve its performance.

Finally, we explored some project ideas and applications where Word2Vec can be used to solve real-world NLP problems.

By now, you should have a solid understanding of how Word2Vec works under the hood and how it can be applied in practice. Word2Vec embeddings have served as a foundation for more advanced NLP techniques and continue to be widely used in the industry.

I hope this post has been informative and helpful in your journey to mastering NLP. Stay tuned for the next part of this series where we will cover more advanced topics and techniques in NLP. Happy learning!