Beyond Bag-of-Words: A Deep Dive into Latent Semantic Analysis for Natural Language Processing

Latent semantic analysis (LSA) is a classical unsupervised learning technique for uncovering the underlying semantic structure of a collection of documents. By learning a linear mapping from the original bag-of-words space to a compact "semantic space", LSA can capture subtle relationships between words and documents that simple keyword matching misses. LSA has been successfully applied to a wide range of natural language processing tasks, from search and recommendation to document classification and essay scoring.

In this post, we‘ll take a detailed look at the mathematical underpinnings of LSA, its relationship to modern embedding techniques, and its applications in real-world natural language processing systems. While LSA has largely been superseded by neural network-based approaches in recent years, it remains an important part of the NLP canon and an instructive case study in the power of representation learning for language. Let‘s dive in!

Mathematical Foundations of LSA

At its core, LSA is a dimensionality reduction technique that seeks to approximate a high-dimensional document-term matrix as the product of several smaller matrices. Specifically, given an n × d document-term matrix M where n is the number of documents and d is the size of the vocabulary, LSA computes a low-rank approximation:

M ≈ U Σ V^T

where:

• U is an n × k orthogonal matrix of left singular vectors
• Σ is a k × k diagonal matrix of singular values
• V is a d × k orthogonal matrix of right singular vectors

The value k is a hyperparameter that controls the dimensionality of the latent semantic space. Setting k << min(n, d) acts as a kind of regularization and noise reduction, forcing the model to discard all but the most salient patterns of word co-occurrence.

Mathematically, the SVD factorization is closely related to both principal component analysis (PCA) and non-negative matrix factorization (NMF). In PCA, the left singular vectors U serve as the principal components, while the singular values Σ are the square roots of the eigenvalues of the covariance matrix. LSA can thus be viewed as a "kernelized" version of PCA applied to a TF-IDF reweighted document-term matrix. The non-negativity constraint in NMF, meanwhile, has been found to encourage more interpretable semantic factors.

It‘s worth noting that naively computing the full SVD of a large matrix can be prohibitively expensive, with cubic running time complexity. However, for k << min(m, n), truncated SVD algorithms like randomized SVD can approximately compute the top k singular vectors and values much more efficiently.

Connections to Neural Embedding Models

LSA was one of the earliest successful embedding models in NLP, representing words and documents as dense vectors that capture their semantic similarity. However, the limitations of bag-of-words models and the linear structure of LSA eventually gave way to nonlinear neural network-based embedding models like word2vec, GloVe, and BERT.

Interestingly, there are some close mathematical connections between LSA and neural language models. For one, the skip-gram word2vec model with negative sampling can be viewed as implicitly factorizing a shifted pointwise mutual information (PMI) matrix, analogous to how LSA factorizes a TF-IDF matrix. Similarly, GloVe‘s log-bilinear model is equivalent to factorizing a log co-occurrence count matrix.

More generally, modern representation learning techniques like autoencoders and self-attention networks can all be viewed as learning compressed encodings of input data that preserve relevant structure. In this sense, LSA can be seen as an early linear autoencoder model for text. The key advantage of deep neural networks is their ability to learn highly nonlinear, hierarchical representations that can capture complex linguistic phenomena.

That said, LSA remains competitive for many applications, especially in specialized domains with small to medium-sized vocabularies. A 2015 study by Altszyler et al., for instance, found that LSA achieved state-of-the-art results on several word similarity benchmarks, outperforming word2vec and GloVe. And in 2020, Cer et al. combined LSA with a pretrained sentence encoder to achieve strong results on sentence-level semantic similarity tasks.

Case Study: Multi-Language Document Classification

To illustrate the power of LSA in a practical setting, let‘s consider the task of multi-language document classification. In this scenario, we have a collection of documents in multiple languages (say English, French, and German) and we want to train a classifier to predict the topic of each document, regardless of its language.

One approach would be to train separate monolingual classifiers for each language using language-specific bag-of-words features. However, this has several drawbacks:

1. It requires labeled training data for each language, which may be expensive or impossible to obtain.
2. It fails to exploit the shared semantic structure across languages, since each classifier is trained independently.
3. It doesn‘t allow for zero-shot cross-lingual transfer to new languages not seen during training.

An elegant alternative is to first embed all documents into a shared multilingual semantic space using LSA, and then train a single classifier on top of the LSA vectors. Specifically, we first construct a multilingual document-term matrix where rows correspond to documents in any language and columns correspond to a unified multilingual vocabulary. We then perform truncated SVD on this matrix to obtain k-dimensional LSA vectors for each document. Finally, we train a classifier (e.g. a support vector machine) on the labeled documents using their LSA vectors as features.

The key advantage of this approach is that it allows the classifier to generalize across languages by capturing the common semantic structure. A 2019 study by Litschko et al. found that an LSA-based classifier achieved over 80% accuracy on a multilingual news topic classification dataset, outperforming monolingual bag-of-words baselines and neural machine translation-based approaches. They further showed that the learned LSA space could be used for zero-shot cross-lingual transfer, achieving over 70% accuracy on new languages like Spanish and Italian without any labeled data.

This case study highlights the power of representation learning to uncover latent structure in multilingual text corpora. By embedding words and documents from different languages into a shared semantic space, LSA enables downstream models to transfer knowledge across languages in a seamless and data-efficient way.

Future Directions and Open Questions

Despite its successes, LSA has several well-known limitations that have driven research into alternative embedding techniques. In particular, LSA struggles to capture polysemy (words with multiple meanings), syntactic structure, and long-range dependencies. More fundamentally, the linearity and bag-of-words assumptions underlying LSA limit its ability to model the full complexity of natural language.

Recent work has explored several directions for extending and improving LSA:

• Tensor factorization methods like Tucker decomposition and canonical polyadic decomposition can directly model higher-order word co-occurrence statistics and account for polysemy.

• Nonlinear dimensionality reduction techniques like t-SNE, UMAP, and autoencoders can learn more expressive latent representations that capture non-linear semantic structure.

• Incorporating external knowledge sources like WordNet, ConceptNet, and knowledge graphs can enrich the learned embeddings and improve interpretability.

• Jointly embedding words, sentences, and documents into a unified semantic space using techniques like doc2vec and skip-thought vectors can enable more flexible transfer learning.

• Adapting LSA to semi-supervised and unsupervised settings using techniques like self-training, co-training, and constrastive learning can reduce the need for expensive labeled data.

Despite these advances, several open questions remain. How can we incorporate linguistic structure and compositionality into latent semantic models? Can we learn cross-modal semantic representations that jointly embed text, images, and other data modalities? How do we ensure the fairness, accountability, and interpretability of learned semantic representations, particularly in high-stakes applications? As the field of representation learning continues to evolve, addressing these questions will be crucial for developing more powerful and responsible NLP systems.

Conclusion

Latent semantic analysis is a powerful technique for uncovering hidden structure in text corpora, with applications ranging from search and recommendation to document classification and cross-lingual transfer learning. By learning a compact linear embedding of words and documents, LSA can capture subtle semantic relationships and enable efficient downstream processing.

At the same time, LSA has its limitations, particularly in modeling non-linear structure, polysemy, and linguistic compositionality. As such, it has largely been supplanted by neural embedding models in many NLP tasks. Nevertheless, LSA remains an important part of the NLP toolkit, particularly in specialized domains with limited data.

More broadly, the success of LSA highlights the power of representation learning for natural language processing. By mapping sparse, high-dimensional text data into compact, semantically-rich vector spaces, embedding models can greatly simplify downstream tasks and enable novel applications. As we continue to develop more sophisticated representation learning techniques, grounded in both linguistic theory and statistical learning, we can expect to see even more impressive advances in NLP in the years to come.