From Word Embeddings to Document Embeddings without Training

In the field of natural language processing (NLP), representing words as dense vector embeddings has revolutionized many tasks like language modeling, text classification, and machine translation. Models like word2vec, GloVe, and fastText learn to map words to real-valued vectors in a way that captures their semantic relationships – words that are similar in meaning end up closer together in the embedding space. For example, the vector for "king" minus the vector for "man" plus the vector for "woman" results in a vector very close to "queen".

While word embeddings are incredibly useful, many NLP tasks require representing entire documents – variable length sequences of words – as fixed-length vectors. Some straightforward approaches include:

1. Simply averaging (or summing) the word vectors in a document. This loses all word order information.

2. Using the element-wise max or min of word vectors. This captures the most salient features but also ignores word order.

3. Concatenating word vectors in order. This preserves word order but results in very high-dimensional representations that can be expensive to work with.

A more sophisticated approach is to weight each word by its importance before combining vectors. A common weighting scheme is tf-idf (term frequency-inverse document frequency). Words that appear frequently in a particular document but rarely in other documents receive higher weight. The idf value captures how informative a word is in general:

idf(w) = log(N / df(w))

where N is the total number of documents and df(w) is the number of documents that contain word w. The tf-idf score is then:

tfidf(w, d) = tf(w, d) * idf(w)

where tf(w, d) is the frequency of word w in document d. We can use these scores to compute a weighted average of word vectors to obtain a document embedding:

v_d = sum(tfidf(w, d) * v_w for w in d) / sum(tfidf(w, d) for w in d)

where v_d is the document vector and v_w is the word vector for word w.

Another approach, known as bag-of-concepts, aims to group semantically similar words into concept clusters. The first step is to cluster the word vectors, for example using the k-means algorithm. Then each document is represented by a vector of concept counts (or concept tf-idfs), with each element corresponding to one of the k concept clusters. This can result in a more compact representation than using all individual word vectors.

It‘s also possible to apply mathematical operations to word vectors before averaging them. Element-wise multiplication can be seen as an AND operation (requiring two words to be similar in a particular dimension), while element-wise max/min correspond to OR operations. Another option is to apply nonlinear functions like the sigmoid to each dimension before averaging.

In kernel methods like support vector machines, data points are implicitly mapped to a high dimensional feature space. We can apply kernel functions to the averaged word vectors to obtain a document embedding in a kernel space. For example, the polynomial kernel of degree p is:

k(v1, v2) = (v1 · v2 + 1)^p

This corresponds to an implicit mapping to a feature space where all monomials up to degree p are considered.

More advanced approaches aim to better capture word order and document-level semantics while still avoiding expensive training procedures:

• Tensor products can be used to bind word vectors together in a way that preserves order. The tensor product v1 ⊗ v2 results in a matrix where each element is a pairwise product of components from v1 and v2. Higher order tensor products v1 ⊗ v2 ⊗ … ⊗ vn can represent a document as an n-dimensional tensor based on all ordered combinations of word vectors. The downside is that the resulting representation can be very high dimensional.

• Autoencoder neural networks can learn to compress the averaged word vectors into a lower dimensional document embedding that preserves the most relevant information. The network is trained to encode an input average word vector and then decode it to produce an output close to the original input. No document labels are needed, making this an unsupervised learning approach.

• Attention mechanisms can be used to focus on the most important words in a document when combining word vectors. A simple approach is to learn attention scores a_w (e.g. a linear function of each word vector) and then compute the document embedding as a weighted average: v_d = sum(a_w * v_w for w in d) / sum(a_w for w in d). The attention scores are normalized to sum to 1 across all words via the softmax function. More sophisticated attention variants like multi-head self-attention have been very successful in recent Transformer language models.

• The Paragraph Vector (PV) model, also known as Doc2Vec, adapts the word2vec framework to learn document embeddings. In PV-DBOW (distributed bag of words), the model learns to predict each word w_i in a document given only the document vector v_d. In PV-DM (distributed memory), the document vector is averaged with the surrounding word vectors and the model learns to predict the next word. Negative sampling is used to keep the computation efficient. While PV requires training on a specific corpus, the learned vectors can be inferred for new documents without re-training.

In practice, pre-computed document embeddings are very convenient as they can be directly used as input features for downstream machine learning models. They are much more computationally efficient than processing the raw text for each new task. The choice of document embedding approach depends on the nature of the task, the size of the document corpus, and the computational resources available.

Simple average word vectors can work surprisingly well and offer a strong baseline. Tf-idf weighting and bag-of-concepts help focus on more informative words. Mathematical operations and kernels can capture some higher order interactions between words. Tensor products and attention mechanisms aim to preserve word order. Paragraph Vectors learn corpus-specific embeddings that can be fine-tuned for different tasks.

On text classification benchmarks, the different document embedding approaches often perform similarly, with more complex methods doing slightly better. Weighted averaging with tf-idf provides consistent improvements over simple averaging. Bag-of-concepts can outperform tf-idf weighting in some cases. Tensor products struggle with high dimensionality and sparsity. Attention-based methods are promising but require more involved training. PV-DBOW is a strong contender when trained on in-domain data.

Some general tips for document embeddings:

• Use pre-trained word vectors like GloVe or FastText as a starting point. These have been trained on very large corpora and capture a lot of semantic information. Consider domain-specific word vectors for specialized topics.

• Normalize word vectors to unit length before applying element-wise operations. This ensures that frequent words don‘t dominate the others.

• Reduce the vocabulary by removing very rare words and corpus-specific stop words. This helps avoid noise from unimportant words.

• Experiment with different dimensionalities for the word vectors and downstream ML models. More dimensions can capture more nuance but increase computational cost.

• For long documents, consider breaking them up into smaller chunks (e.g. paragraphs) and averaging those embeddings. This provides more homogeneous semantic units.

• When faced with novel words or misspellings, try to match them to the closest word in the vocabulary. Word vectors are fairly robust to typos.

• Consider ensembling different types of document embeddings as input to ML models. They can capture complementary information.

Looking ahead, an important direction for future research is to develop document embedding methods that better capture the high-level semantic structure of documents in a fully unsupervised way, without relying on word vectors pre-trained for a different task. This could potentially leverage advances in language model pre-training and unsupervised sentence embeddings.

In summary, going from word embeddings to document embeddings without additional training is a challenging but important task in NLP. A variety of approaches based on weighting, clustering, mathematical operations, and attention mechanisms have been proposed. Picking the right method depends on the downstream task and available resources. Pre-computed document embeddings offer an efficient way to harness the power of large unlabeled text corpora for practical applications.