A Deep Dive into Text Vectorization for NLP: An Expert Guide

Text vectorization, or embedding words and documents into a numeric vector space, is a key step in natural language processing (NLP). It enables machine learning models to extract rich semantic information from unstructured text. Over the years, techniques have evolved from simple frequency-based methods to complex neural language models. In this guide, we‘ll thoroughly examine these approaches from an AI/ML expert perspective.

Why Text Vectorization is Essential for NLP

NLP models require text to be transformed into numeric vectors for processing. The goal is to represent words or documents as dense vectors in a high-dimensional space, where semantically similar entities are mapped to nearby points. This vector representation enables ML models to efficently learn useful patterns from text data.

Text vectorization is crucial for tasks like:

• Document classification and clustering
• Sentiment analysis and opinion mining
• Named entity recognition and information extraction
• Machine translation and text summarization
• Question answering and information retrieval

The choice of vectorization technique can greatly impact model performance. Let‘s look at key methods, from traditional statistical approaches to state-of-the-art neural embeddings.

Statistical Text Vectorization Methods

Traditional methods represent text as sparse vectors capturing word frequency statistics. While simple, they form the foundation for text representation.

Bag-of-Words (BoW) Model

The BoW model represents a document as an unordered collection of word frequencies. Given a vocabulary V of unique words, a document is a vector in ℝ^|V| where each element is the count of a word.

Formally, for a document d and a word w:

BoW(d, w) = f(w, d)

where f(w, d) is the frequency of word w in document d.

For example:

• d1 = "John likes to watch movies. Mary likes movies too."
• d2 = "John also likes to watch football games."

With vocabulary V = [John, likes, to, watch, movies, Mary, too, also, football, games], the BoW vectors are:

BoW(d1) = [1, 2, 1, 1, 2, 1, 1, 0, 0, 0] BoW(d2) = [1, 1, 1, 1, 0, 0, 0, 1, 1, 1]

BoW is simple and intuitive but has limitations:

• Ignores word order and grammar
• Suffers from data sparsity and high dimensionality
• Equally weights common and rare words

BoW vectors have dimensionality |V| which grows with corpus size. Analysis shows BoW produces 99\%+ sparse vectors for real-world corpora Dhillon & Ungar 2011.

TF-IDF Model

TF-IDF extends BoW by weighting word frequencies by their inverse document frequency. TF captures word importance within a document, while IDF captures word specificity across the corpus.

For word w in document d from corpus D:

TF(w, d) = f(w, d) / Σ_i f(w_i, d)

IDF(w, D) = log [ |D| / Σ_{d∈D} 1[w ∈ d] ]

TF-IDF(w, d, D) = TF(w, d) · IDF(w, D)

where f(w, d) is the raw count, |D| is the corpus size, and 1[·] is the indicator function.

Intuitively:

• Words frequent in a document get high TF
• Words appearing in many documents get low IDF
• TF-IDF highlights words that are frequent in a document but rare overall

A 2019 study found TF-IDF remains a strong baseline, beating many newer methods on topic classification tasks Usai et al. 2019.

N-Gram Models

N-grams extend BoW to capture local word order and phrases. An n-gram is a contiguous sequence of n words. The vector space is all unique n-grams in the corpus.

For example, the bigrams (n=2) of "John likes to watch movies" are:

[(‘John‘, ‘likes‘), (‘likes‘, ‘to‘), (‘to‘, ‘watch‘), (‘watch‘, ‘movies‘)]

The bigram vector has a dimension for each unique bigram and values are bigram frequencies.

N-grams help model word order but exacerbate sparsity and dimensionality issues. In practice, a combination of unigrams, bigrams, and trigrams often works best.

While statistical methods are interpretable and efficient, they struggle to capture rich semantics. Neural embeddings aim to overcome these limitations.

Neural Text Embedding Methods

Neural networks can learn dense, low-dimensional word and document vectors that capture semantic relationships. These vectors are learned end-to-end from data.

word2vec Model

Word2vec is a framework for learning word embeddings using shallow neural networks Mikolov et al. 2013. It comes in two flavors:

• Continuous bag-of-words (CBOW): predicts a word given its context
• Skip-gram: predicts the context given a word


The network is trained to optimize the objective:

J(θ) = Σ_{w∈V} [ log P(w | context(w)) ]

where context(w) is the set of words in a fixed-size window around w.

After training, the learned weights of the hidden layer are the word embeddings. Words with similar contexts end up with similar vectors.

Key highlights of word2vec:

• Produces dense, fixed-length vectors (usually 100-1000 dimensions)
• Vectors capture semantic and syntactic regularities
• Efficient to train on large corpora
• Pre-trained embeddings readily available (Google News, Wikipedia, Twitter)

On the analogy task of a is to b as c is to ?, word2vec scores 74\% accuracy Mikolov et al. 2013.

GloVe Model

Global Vectors (GloVe) learns word vectors by factorizing the global word-word co-occurrence matrix Pennington et al. 2014. Compared to word2vec which trains on separate local contexts, GloVe captures global corpus statistics.

The co-occurrence matrix X, where X_ij is the number of times word i appears in the context of word j, is factored to a product of lower dimensional matrices:

X ≈ W · C^T

where W and C are |V| × d word and context matrices. The GloVe objective is:

J(θ) = Σ_i Σ_j f(X_ij) (w_i^T c_j + b_i + b_j – log(X_ij))^2

where f(X_ij) is a weighting function to give more importance to frequent co-occurrences.

The learned W matrix gives the word embeddings. GloVe outperforms word2vec on several benchmark tasks including word analogy, word similarity, and named entity recognition Pennington et al. 2014.

fastText Model

FastText extends word2vec by representing words as character n-grams Bojanowski et al. 2017. Each word vector is the sum of its character n-gram vectors:

v(w) = Σ_{g∈G_w} z_g

where G_w is the set of character n-grams in word w and z_g is the vector for n-gram g.

This allows computing good vectors for rare or out-of-vocabulary words by leveraging subword information. It‘s especially useful for morphologically rich languages.

FastText matches the performance of word2vec with a huge reduction in model size. It works well on datasets with many rare words.

Contextual Word Embeddings

The above methods learn a single vector per word, ignoring context and polysemy. Contextual embeddings instead generate dynamic word vectors based on surrounding context.

ELMo (Embeddings from Language Models) uses a deep bidirectional LSTM trained on a language modeling objective Peters et al. 2018. ELMo word vectors are a learned combination of the internal states of the LSTM.

BERT (Bidirectional Encoder Representations from Transformers) uses a transformer encoder pretrained on masked language modeling Devlin et al. 2019. BERT attends to both left and right context when generating word vectors.

For the sentences:

• The bank will close your account if you have insufficient funds.
• We had a picnic on the bank of the river.

ELMo and BERT would generate different vectors for "bank", capturing the "financial institution" vs "river bank" meaning based on context. Static embeddings would give the same vector.

Contextual embeddings are the state-of-the-art on many NLP tasks. The downside is high computational cost – BERT has 340M parameters compared to 3M for word2vec.

Applications and Practical Considerations

The choice of vectorization method depends on your data, task, and compute budget. Some general guidelines:

• Bag-of-words and TF-IDF are good for fast prototyping or when interpretability is key. Use n-grams when local word order matters.

• Word2vec, GloVe, and fastText are solid choices for most tasks. They balance performance and efficiency. FastText shines for rare words and morphologically rich languages.

• If you have complex tasks like question answering or machine translation, use ELMo or BERT embeddings. They give top performance at the cost of model size and speed.

• Consider the language and domain when choosing embeddings. For niche domains, train your own embeddings on domain-specific text.

Always benchmark multiple approaches and use the simplest model that gives good enough results. The field evolves quickly so keep experimenting with new techniques.

Conclusion

Text vectorization is a key component of modern NLP. We‘ve traced its evolution from simple frequency-based methods to contextualized neural language models.

Frequency methods like bag-of-words and TF-IDF are intuitive and fast but struggle with sparsity and semantics. Neural embeddings like word2vec, GloVe, and fastText learn dense semantic vectors but have a single representation per word. Contextual models like ELMo and BERT generate dynamic vectors based on surrounding text, giving state-of-the-art results at the cost of complexity.

When choosing a vectorization approach, consider your task requirements, dataset characteristics, and compute budget. Benchmark multiple methods and use the simplest one that performs well.

Text vectorization is a complex and rapidly evolving field. But a strong grasp of classical and modern techniques is essential for anyone working in NLP. I hope this deep dive has given you valuable insights to apply in your own projects. Remember, the key is turning unstructured text into meaningful numeric representations — choose your techniques wisely!