Part 15: Step by Step Guide to Master NLP – Topic Modeling using NMF

Introduction

Welcome back to our multi-part series on natural language processing! In the last installment, we took a high-level look at topic modeling – a powerful technique for automatically discovering the hidden thematic structure within a collection of documents. Today, we‘ll be diving deep into one specific topic modeling algorithm that has become popular in recent years: Non-negative Matrix Factorization, or NMF for short.

Topic modeling has numerous applications across domains as diverse as:

• Content recommendation systems
• Scientific literature exploration
• Social media analysis
• Healthcare record mining
• Fraud detection
• Analyzing product reviews

But how exactly does a computer algorithm "discover" abstract topics within raw text documents? The key lies in the fascinating field of linear algebra and matrix factorization. Get ready to geek out on some math!

Non-negative Matrix Factorization 101

At its core, NMF is a linear algebra technique that falls under the broad umbrella of dimension reduction. It operates on the basic assumption that a set of documents can be represented as a matrix – with each document as a row and the frequencies of every unique word in the vocabulary as columns.

The goal is to approximately factor this large matrix V (typically very sparse with lots of zero values since most documents only contain a small subset of words) into the product of two smaller non-negative matrices W and H, such that:

V ≈ W × H

Intuitively, you can think of the W matrix as representing topics, with each column corresponding to a cluster of semantically related words that form a coherent theme. The H matrix captures the proportions or weights of those topics within each original document.

Critically, both the W and H matrices are constrained to only contain non-negative values (hence the "non-negative" in NMF). This non-negativity constraint allows for better interpretability of the factors compared to other techniques like SVD that result in negative coefficients.

A Concrete Example

Let‘s say we have a toy dataset of 1000 movie reviews. After preprocessing the text to remove stopwords and rare terms, we‘re left with a vocabulary of 5000 unique words. So our input matrix V would have dimensions 1000 x 5000.

If we want to discover the 10 main topics/themes that the reviews tend to discuss, we can run NMF to decompose it into:

V ≈ W × H, where W is 1000 x 10 and H is 10 x 5000

Each of the 10 columns in W will contain weights associating the original 5000 terms with that topic. The top weighted terms give us a good sense of what the topic represents. For example, we might see themes like:

• Topic 1: "acting", "actor", "role", "performance", "played"
• Topic 2: "effects", "CGI", "visuals", "graphics", "3D"
• Topic 3: "plot", "story", "writing", "script", "premise"

Each row in the H matrix will tell us the proportions of the 10 topics present in the corresponding review, serving as a lower-dimensional "summary" of its content.

We can even push this further and cluster the movie reviews based on their topic distributions – uncovering communities of reviews that share similar themes and opinions!

The Math Behind the Magic

Alright, time to put on our mathematician hats and really scrutinize what‘s going on under the hood in NMF. The fundamental idea is to cast the matrix factorization as an optimization problem. We start with some initial random guesses for W and H and then iteratively refine them to minimize a cost function capturing the error between the true matrix V and our current approximation W × H.

There are different cost functions used in NMF, but two of the most common are:

Frobenius Norm

Formally, this minimizes the squared error between the elements of the original matrix V and the approximation W × H:

minimize ||V – W × H||²

The double bars represent the Frobenius norm, which is essentially just the square root of the sum of squared elements. This is the classic "least squares" approach also used in linear regression. Python‘s numpy module provides a built-in function to compute it:

import numpy as np
error = np.linalg.norm(V – W @ H)

Kullback-Leibler Divergence

An alternative approach is to minimize the KL divergence between V and W × H, which is a measure of the difference between two probability distributions:

minimize ∑[V × log(V / W × H) – V + W × H]

Where ∑ represents the sum of all elements. KL divergence is frequently used for problems dealing with probabilities/proportions. You can implement it in numpy as:

error = np.sum(V * np.log(V / (W @ H)) – V + W @ H)

Update Rules

Here‘s the really cool part – no matter which cost function we choose, we can derive simple multiplicative update rules for W and H that are guaranteed to monotonically decrease the optimization objective at each iteration!

For Frobenius norm, the updates work out to:
W = W × (V × Hᵀ) ÷ (W × H × Hᵀ)
H = H × (Wᵀ × V) ÷ (Wᵀ × W × H)

Where × represents matrix multiplication, ÷ is element-wise division, and ᵀ is the transpose operation. The proof is a bit involved, but these remarkably simple equations do the job!

We just keep alternating updates of W and H until the cost function converges or we reach a maximum number of iterations. The resulting factors W and H give us an approximate factorization of the original matrix V.

Practical Considerations

In the real world, NMF requires some care to get the best results. Here are a few tips and tricks to keep in mind:

Initialization

Since the alternating updates can get stuck in local optima, the initial values of W and H matter. Some popular initialization schemes are:

• Nonnegative SVD – Initialize using the factors from a SVD decomposition, clipping negative values to 0
• Random – Initialize with small random non-negative values
• Clustering – Initialize W to the cluster centroids from running k-means on the documents

Choosing Number of Topics

NMF requires you to specify the number of topics k in advance, which can be tricky. Some heuristics include:

• Coherence scores – Measure the semantic similarity of top words in each topic and choose k to maximize coherence
• Topic specificity – Track how the top words change as k increases and stop when they become too broad
• Domain knowledge – Leverage expertise about the expected number of themes in the dataset

In practice, it‘s good to experiment with a range of values and qualitatively examine the results.

Normalization and Smoothing

NMF is quite sensitive to the scale of the input data. It‘s usually beneficial to normalize the rows of V to sum to 1, effectively interpreting the matrix as word proportions per document rather than raw counts. Adding small smoothing constants to V can also help avoid divide-by-zero issues.

Applications and Extensions

We‘ve barely scratched the surface of what‘s possible with NMF! Let‘s look at a few more applications and variants that data scientists are using:

Image Feature Extraction

One cool use case is in image processing. If we construct V so that each row is a vectorized grayscale image, the columns of W become "basis images" and H captures how much of each basis is used to reconstruct an image. By keeping k small, NMF performs a sort of lossy image compression.

Time Series Analysis

NMF can also be used to discover temporal patterns in time series data. If we slice a multivariate time series into fixed-length windows and stack them as rows of V, the resulting topics in W will represent common subsequences or "motifs."

Incorporating Side Info

There have been many extensions to NMF that incorporate additional side information alongside the raw document-word matrix. For example, adding regularization terms to the cost function can encourage topics to align with document metadata like timestamps or author information.

Deep Learning Models

Modern neural approaches to topic modeling like DocNADE combine ideas from probabilistic topic models with deep autoregressive architectures. These models can learn more complex, nonlinear topic representations and scale to massive datasets.

Conclusion and Next Steps

Phew, that was a lot to take in! We‘ve gone from the high-level intuition behind NMF to the nitty-gritty of the math and optimization. I hope you‘ve come away with an appreciation for the power and flexibility of this technique.

If you‘re raring to go try it out yourself, I recommend starting with some of these beginner-friendly projects:

• Analyzing the State of the Union addresses
• Topic modeling a collection of Wikipedia articles
• Building a content recommendation system using NMF

I‘ve put together a GitHub repo with code to get you started. Remember, the best way to really understand this stuff is to get your hands dirty and experiment!

Looking ahead, an exciting direction in topic modeling research is the integration of neural language models like BERT. By using the contextual word embeddings from a pretrained language model, we can potentially learn even richer topic representations that capture subtle shades of meaning.

I hope you‘ve enjoyed this deep dive into NMF for topic modeling. Stay tuned for the next installment in this series, where we‘ll be looking at word embeddings and how they‘ve revolutionized NLP.

As always, if you have any questions or just want to geek out about NLP, feel free to drop me a line! You can find my contact info and social media handles in the author bio below. Until next time, happy modeling!