Topic Modeling with LDA: A Hands-On Introduction

ByJordan Brown

Whether you‘re working with customer reviews, news articles, social media posts, or any other text data, being able to automatically discover the latent topics in a large collection of documents is an essential skill for any data scientist or analyst. Topic modeling provides a way to organize, understand, and summarize large volumes of text by identifying the hidden semantic structures within the data.

In this post, we‘ll dive into topic modeling using Latent Dirichlet Allocation (LDA), a powerful algorithm that has become synonymous with topic modeling. We‘ll explain what LDA is, walk through how it works step-by-step, and illustrate it with an example. Then we‘ll apply LDA to perform topic modeling on a real dataset using Python. By the end, you‘ll have a solid understanding of topic modeling with LDA and be able to apply it to your own projects.

What is Topic Modeling?

In a nutshell, topic modeling is a type of statistical modeling for discovering the abstract "topics" that occur in a collection of documents. It can be thought of as a form of text mining – a way to obtain recurring patterns of words in textual material.

The key idea is that any document exhibits multiple topics, in different proportions. For example, a news article about a new Apple iPhone might be 60% about "technology" and 40% about "business". And a research paper on deep learning could be 50% about "machine learning", 30% about "neural networks", and 20% about "optimization".

With a large enough collection of documents, topic modeling techniques aim to automatically discover these latent topics based on the statistical distributions of words across the documents. The topics are usually represented as a set of most relevant keywords.

Topic modeling is an unsupervised approach, meaning it doesn‘t require any prior annotations or labeling of the documents – the topics emerge from the analysis of the original texts. This is in contrast to supervised models like those used for document classification, which rely on manually labeled training data.

Applications of Topic Modeling

Some common applications and use cases of topic modeling include:

  • Content recommendation systems – identifying topics a user is interested in based on their reading history and suggesting similar content
  • Document summarization – automatically producing summaries of large documents by extracting a few sentences that best represent the major topics
  • Customer feedback analysis – discovering common themes and pain points from surveys and reviews, without manual reading
  • Research literature exploration – understanding the main subject areas in a large corpus of scientific papers
  • Spam filtering – detecting unwanted advertising or inappropriate content in user-generated text
  • Trend tracking – following the evolution of societal, cultural, or political topics over time on social media

In general, topic modeling can be applied to any domain where understanding text is important and the volume of text is large enough that manually reading it all is impractical.

Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) is a particularly popular algorithm for topic modeling. It‘s a generative probabilistic model that assumes each document can be described as a mixture of a fixed number of topics and that each word‘s presence is attributable to one of those topics.

LDA was first presented by David Blei, Andrew Ng, and Michael I. Jordan in 2003 in their paper "Latent Dirichlet Allocation". It has since became one of the most widely used techniques for topic modeling and has been extended and adapted in various ways.

The "Dirichlet" in LDA refers to the Dirichlet distribution, which is a distribution over distributions. In LDA, the topic distribution per document and the word distribution per topic are assumed to have Dirichlet priors.

How LDA Works

At a high level, LDA represents documents as mixtures of topics that spit out words with certain probabilities. It assumes that documents are produced in the following fashion:

  1. Decide on the number of words N the document will have
  2. Choose a topic mixture for the document (according to a Dirichlet distribution over a fixed set of K topics). For example, assuming that we have the two food-related topics, 90% of the document might be generated from the first topic and 10% from the second.
  3. Generate each word in the document by:
  • First picking a topic (according to the multinomial distribution that you sampled above; for example, you might pick the first topic with 90% probability and the second topic with 10% probability).
  • Using the topic to generate the word itself (according to the topic‘s multinomial distribution). For example, the food-related topics will have words about food with higher probability.

Assuming this generative model for a collection of documents, LDA then tries to backtrack from the documents to find a set of topics that are likely to have generated the collection.

LDA Algorithm Details

More formally, the goal of LDA is to maximize the following likelihood function:

$p(D|\alpha,\beta) = \prod_{d=1}^M \int p(\thetad|\alpha) \left( \prod{n=1}^{Nd} \sum{z{dn}} p(z{dn}|\thetad)p(w{dn}|z_{dn},\beta) \right) d\theta_d$

where:

  • $D = {w_1, w_2, \cdots, w_M}$ is the corpus
  • $\alpha$ is the parameter of the Dirichlet prior on the per-document topic distributions
  • $\beta$ is the parameter of the Dirichlet prior on the per-topic word distribution
  • $\theta_d$ is the topic distribution for document $d$
  • $z_{dn}$ is the topic for the $n$-th word in document $d$
  • $w_{dn}$ is the $n$-th word in document $d$

Since this likelihood function is intractable due to the coupling between $\theta$ and $\beta$, the solution is to use approximate inference techniques like variational inference or Gibbs sampling.

The key inferential problem that LDA is trying to solve is that of computing the posterior distribution of the hidden variables given a document:

$p(\theta, \mathbf{z}|\mathbf{w},\alpha,\beta)$

i.e. the distribution of the topic structure $\theta$ and assignments $\mathbf{z}$ given the observed words $\mathbf{w}$ and model parameters $\alpha$ and $\beta$. This distribution is intractable to compute in general, but a variety of approximate inference techniques have been used, including Laplace approximation, variational approximation, and Markov chain Monte Carlo.

Illustrative Example

To make the LDA process more concrete, let‘s consider a simple example. Suppose we have the following set of documents:

  1. I love eating broccoli and bananas.
  2. I ate a banana and spinach smoothie for breakfast.
  3. Chinchillas and kittens are cute.
  4. My sister adopted a kitten yesterday.
  5. Look at this cute hamster munching on a piece of broccoli.

Intuitively, by reading through these documents, we can see two distinct topics: one about food and one about cute animals. The goal of LDA is to automatically discover these latent topics.

Following the generative process we described earlier, LDA might produce something like:

  • Topic 1: 30% broccoli, 15% bananas, 10% breakfast, 10% munching, … (topic about food)
  • Topic 2: 20% chinchillas, 20% kittens, 20% cute, 15% hamster, … (topic about cute animals)

Then for each document:

  1. 100% Topic 1
  2. 100% Topic 1
  3. 100% Topic 2
  4. 100% Topic 2
  5. 60% Topic 2, 40% Topic 1

Implementing LDA in Python

Now that we understand how LDA works, let‘s see how we can implement it in Python. We‘ll use the gensim library, which is specifically designed for topic modeling.

First, we need to install gensim:

pip install gensim

Next, let‘s prepare our text data. We‘ll use a dataset of NPR news article excerpts, which you can find on Kaggle: https://www.kaggle.com/datasets/rtatman/national-public-radio-nlp-data

Preprocessing

Before we can apply LDA, we need to preprocess the text data. This involves:

  1. Tokenization – splitting the text into individual words
  2. Removing stopwords – getting rid of common words like "the", "a", "in", etc.
  3. Lemmatization – converting words to their dictionary form (e.g., "running" -> "run")

Here‘s how we can do this using the NLTK library:

import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

stop_words = stopwords.words(‘english‘)
lemmatizer = WordNetLemmatizer()

def preprocess(text):
    result = []
    for token in gensim.utils.simple_preprocess(text):
        if token not in stop_words and len(token) > 3:
            result.append(lemmatizer.lemmatize(token))
    return result

Now let‘s read in our data and preprocess it:

import pandas as pd

npr = pd.read_csv(‘npr.csv‘)
npr[‘article_text‘] = npr[‘Article‘].apply(preprocess)

Creating the Dictionary and Corpus

Next, we need to create a dictionary from the preprocessed data. The dictionary maps every word to a unique integer ID.

dictionary = gensim.corpora.Dictionary(npr[‘article_text‘])

We can filter out words that occur very frequently or very rarely to reduce noise:

dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)

Next, we create the corpus, which is a list of bags of words. Each bag of words is a list of (token_id, token_count) tuples.

corpus = [dictionary.doc2bow(text) for text in npr[‘article_text‘]]

Training the LDA Model

Now we‘re ready to train our LDA model:

from gensim.models import LdaMulticore

lda_model = LdaMulticore(corpus=corpus,
                         id2word=dictionary,
                         num_topics=10)

Here we‘re specifying that we want to find 10 topics. You can adjust this parameter based on your needs and the size of your dataset.

Examining the Topics

Once the model is trained, we can print out the topics:

from pprint import pprint

pprint(lda_model.print_topics())

This will give us output like:

[(0, ‘0.028*"music" + 0.020*"band" + 0.012*"album" + 0.009*"song" + 0.008*"jazz"‘),
(1, ‘0.019*"president" + 0.015*"trump" + 0.009*"government" + 0.007*"war" + 0.007*"county"‘),
(2, ‘0.016*"child" + 0.013*"school" + 0.010*"student" + 0.009*"family" + 0.009*"parent"‘),
...
]

Each topic is represented as a combination of keywords and each keyword‘s contribution to the topic. These keywords give us insight into what the topic might represent.

We can also get the topic distribution for a specific document:

doc_lda = lda_model[corpus[0]]
print(doc_lda)

This will give us output like:

[(0, 0.020001831829864054),
(1, 0.02000204822465949),
(2, 0.020002028062953602),
(3, 0.49631042031583406),
(4, 0.020002028428188243),
(5, 0.020001696638191544),
(6, 0.020001748318210017),
(7, 0.020001944299210224),
(8, 0.020001297312698683),
(9, 0.3433470964908393)]

Each tuple represents the topic and its proportion for this particular document.

Alternative Approaches

While LDA is one of the most popular methods for topic modeling, it‘s not the only one. Other common techniques include:

  • Latent Semantic Analysis (LSA) – Uses singular value decomposition to find latent topics. It doesn‘t have a probabilistic interpretation like LDA.

  • Non-Negative Matrix Factorization (NMF) – Factorizes the document-term matrix into a document-topic matrix and a topic-term matrix, with the constraint that all matrices have no negative elements.

  • Hierarchical Dirichlet Process (HDP) – An extension of LDA that automatically determines the number of topics.

Each method has its own strengths and weaknesses, and the choice of method often depends on the specific dataset and application.

Conclusion

In this post, we‘ve covered the basics of topic modeling and explored Latent Dirichlet Allocation (LDA) in depth. We‘ve seen how LDA can automatically discover latent topics in a collection of documents, and we‘ve implemented it in Python using the gensim library.

Topic modeling is a powerful technique for making sense of large volumes of unstructured text data. It has numerous applications, from content recommendation to trend analysis. By understanding how topic modeling works and being able to apply it in practice, you can gain valuable insights from your text data.

Further Reading

If you want to dive deeper into topic modeling and LDA, here are some resources to check out:

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts