Part 18: Step by Step Guide to Master NLP – Topic Modelling using LDA (Probabilistic Approach)

Topic modeling is a powerful technique in natural language processing (NLP) for discovering the hidden thematic structure in large collections of documents. It allows us to extract abstract "topics" that occur in a collection, where each topic is represented by a distribution over words. Topic models have diverse applications including content recommendation, document clustering, information retrieval, and feature engineering for downstream NLP tasks.

Latent Dirichlet Allocation (LDA) is one of the most popular and widely used topic modeling algorithms. It is a probabilistic generative model that aims to explain the observed words in documents by inferring latent topics. In this post, we‘ll dive deep into the inner workings of LDA from a probabilistic perspective. We‘ll build up the intuition behind the model, walk through the generative process and key probabilistic assumptions, and see how to implement it in Python. Let‘s get started!

Intuition behind LDA

The basic idea behind LDA is that documents exhibit multiple topics, where each document can be viewed as a mixture of various topics and each topic is characterized by a distribution over words. For example, a news article about a new space mission might combine topics like "space exploration", "science", and "politics".

More formally, LDA assumes that each document is generated by the following process:

1. Choose the number of words N the document will have (say, according to a Poisson distribution).
2. Choose a topic mixture for the document (according to a Dirichlet distribution over a fixed set of K topics). For example, the document might consist of 1/3 space, 1/3 science, and 1/3 politics.
3. Generate each word in the document by:
   a) First picking a topic (according to the document‘s multinomial distribution over topics).
   b) Then using the topic to generate the word according to the topic‘s multinomial distribution over the vocabulary.

Assuming this generative model for a collection of documents, LDA then tries to infer the latent topics that are most likely to have generated the observed collection.

Mathematical Formulation

Now let‘s define the key mathematical concepts more precisely.

In LDA, each document is represented as a bag-of-words, disregarding grammar and word order. Given a corpus of M documents and a vocabulary of V words, let:

• w_ij = the j‘th word in the i‘th document
• z_ij = the topic for the j‘th word in the i‘th document
• theta_i = topic distribution for document i
• phi_k = word distribution for topic k
• alpha = Dirichlet parameter on the per-document topic distributions
• beta = Dirichlet parameter on the per-topic word distributions

Then LDA specifies the following generative process for each document w in a corpus D:

1. Choose N_i ~ Poisson(xi)
2. Choose theta_i ~ Dirichlet(alpha)
3. For each of the N_i words w_ij:
   • Choose a topic z_ij ~ Multinomial(theta_i)
   • Choose a word w_ij ~ Multinomial(phi_zij)

The goal is then to infer the latent variables that most likely generated the observed documents, namely:

• The topics z
• The topic distributions theta
• The topic-word distributions phi

The posterior distribution of the latent variables given the documents P(z,theta, phi|w, alpha, beta) cannot be computed exactly, but efficient approximate inference techniques like variational inference or Gibbs sampling can be used.

Dirichlet Distribution

One of the key assumptions in LDA is that both the topic distribution theta for each document and the word distributions phi for each topic follow Dirichlet distributions.

The Dirichlet distribution Dir(alpha) is a family of continuous multivariate probability distributions parameterized by a vector alpha of positive reals. It is the conjugate prior distribution for the parameters of the multinomial distribution, which is why it is a natural choice for modeling the topic and word distributions in LDA.

The Dirichlet distribution is often used as a prior distribution in Bayesian statistics because it is flexible and has some nice properties. The shape of the distribution is determined by the concentration parameter alpha:

• If alpha < 1, the distribution is sparse, generating samples that are highly concentrated around a few components.
• If alpha = 1, we get a uniform distribution over the simplex.
• If alpha > 1, the distribution becomes more dense, with samples more evenly spread across the components.

For topic modeling, a symmetric Dirichlet distribution with a concentration parameter alpha < 1 is commonly used as a sparse prior for the per-document topic distributions theta. This encodes our prior belief that each document likely covers just a few topics rather than all topics equally. Likewise, a symmetric Dirichlet prior with beta < 1 is used for the per-topic word distributions phi.

Training LDA with Collapsed Gibbs Sampling

Given a document corpus, how do we actually learn the latent topic structure? There are a few different algorithms that are used to train LDA, including variational inference and Markov chain Monte Carlo (MCMC) methods. Here we‘ll walk through collapsed Gibbs sampling, which is a popular MCMC approach.

The idea is to iteratively sample each latent topic z_ij for each word, conditioned on the current values of all the other latent topics. After a number of iterations, the sampler will converge to the posterior distribution P(z,theta,phi|w, alpha, beta) and we can estimate the topic distributions theta and word distributions phi from the samples.

The key steps are:

1. Initialize the topics z randomly for all words in the corpus

2. Repeat until convergence:

   • For each document i in 1…M:

     • For each word j in 1…N_i:

       • Exclude z_ij from the topic counts and word counts

       • Sample a new topic z_ij from the conditional multinomial P(z_ij | z_-ij, w) given by:

         P(z_ij=k | z_-ij, w) ∝ (n_k,-i⁽ⁱ⁾ + alpha) (n_wij,-ij^(k) + beta) / (n_.^(k) + Vbeta)

         where n_k,-i⁽ⁱ⁾ is the number of words in document i assigned to topic k, excluding the current word w_ij.
         n_wij,-ij^(k) is the number of times word w_ij is assigned to topic k, excluding the current instance.
         n_.^(k) is the total number of words assigned to topic k.

3. Estimate the topic-word distributions phi and document-topic distributions theta as:

   phi_kw = (n_w^(k) + beta) / (n_.^(k) + V * beta)

   theta_ik = (n_k⁽ⁱ⁾ + alpha) / (n_.⁽ⁱ⁾ + K * alpha)

The conditional multinomial for sampling z_ij has an intuitive interpretation. The first term (n_k,-i⁽ⁱ⁾ + alpha) represents the probability of topic k in document i, based on the topic counts for that document. The second term (n_wij,-ij^(k) + beta) / (n_.^(k) + V*beta) represents the probability of word w_ij under topic k, based on the word counts for that topic. So the new topic is sampled proportional to both how prevalent that topic is in the current document, and how much that topic seems to like the current word, based on the co-occurrence counts across the corpus.

Implementing LDA in Python with Gensim

Gensim is a popular Python library for topic modeling that makes it easy to implement LDA. Here‘s a simplified example of how to train an LDA model on a toy corpus:

from gensim import corpora, models

# Sample document corpus
texts = [
    [‘space‘, ‘nasa‘, ‘moon‘, ‘earth‘],
    [‘genes‘, ‘dna‘, ‘genome‘, ‘sequencing‘, ‘dna‘],
    [‘politics‘, ‘president‘, ‘state‘, ‘government‘, ‘law‘]
]

# Create a dictionary from the corpus
dictionary = corpora.Dictionary(texts)

# Convert the corpus into a bag-of-words representation
corpus = [dictionary.doc2bow(text) for text in texts]

# Train an LDA model
lda_model = models.LdaModel(corpus=corpus, id2word=dictionary, num_topics=3)

# Print the topics and top words
print(lda_model.print_topics())

This would output something like:

[(0, ‘0.029*"dna" + 0.029*"genes" + 0.029*"genome" + 0.029*"sequencing"‘), 
 (1, ‘0.029*"state" + 0.029*"president" + 0.029*"government" + 0.029*"politics"‘),
 (2, ‘0.029*"earth" + 0.029*"moon" + 0.029*"nasa" + 0.029*"space"‘)]

Here we can see that LDA has inferred 3 topics from the small corpus, and the top words for each topic correspond to the themes of genetics, politics, and space.

Of course, this is just a toy example – in practice you would train on a much larger corpus, preprocess the text (tokenization, remove stop words, etc), and tune the model hyperparameters like num_topics, alpha and beta.

Tips and Best Practices

Here are a few tips and best practices to keep in mind when working with LDA:

• Preprocess the text appropriately. This includes tokenization, lowercasing, removing punctuation, removing stop words, and stemming/lemmatization. Gensim provides utilities for these steps.

• Experiment with different numbers of topics K. Choosing the right K depends on the size and nature of your corpus and your application. Perplexity and coherence scores can help assess topic quality.

• Try different settings of the hyperparameters alpha and beta. In general, alpha controls the sparsity of the document-topic distributions (lower alpha -> sparser distributions) and beta controls the sparsity of the topic-word distributions (lower beta -> sparser distributions).

• Look at the top words for each topic and see if they are meaningful and coherent. You may need to adjust the preprocessing, number of topics, or hyperparameters if the topics don‘t make sense.

• Be aware that LDA has some limitations and assumptions that may not always hold in practice, such as the bag-of-words assumption and the fixed number of topics. There are many extensions and alternatives to LDA that can relax these assumptions, such as Hierarchical Dirichlet Process, Correlated Topic Models, and structural topic models.

Conclusion

In this post, we took a deep dive into Latent Dirichlet Allocation from a probabilistic perspective. We explored the generative process behind the model, the role of the Dirichlet distribution, and how to train LDA using collapsed Gibbs sampling. We also saw how to easily implement LDA in Python using the Gensim library.

Topic models like LDA are a powerful tool for uncovering the hidden thematic structure in text corpora, and have numerous applications across NLP, information retrieval, and the digital humanities. While we focused on the intuition and mechanics here, LDA is supported by elegant probabilistic foundations and has deep connections to Bayesian inference, graphical models, and information theory.

I encourage you to try out LDA on your own datasets, experiment with the hyperparameters, and think critically about the assumptions and interpretations. Feel free to reach out with any questions, feedback, or insights!