# A Comprehensive Guide to Key Phrase Extraction with TF\-IDF in Python

- Canonical: https://33rdsquare.com/how-to-extract-key-phrases-using-tfidf-with-python/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Key phrase extraction is an important technique in natural language processing for automatically identifying the main topics and themes in a piece of text. By distilling a document down to its most relevant and representative phrases, key phrase extraction enables tasks like document summarization, topic modeling, and semantic search.

One of the most popular and effective methods for extracting key phrases is TF-IDF, which stands for Term Frequency – Inverse Document Frequency. In this in-depth tutorial, we‘ll take a close look at how TF-IDF works and walk through a complete example in Python of using it to extract key phrases from a dataset of documents.

By the end of this guide, you‘ll have a solid understanding of TF-IDF and the tools to apply it to your own projects. Let‘s dive in!

## Understanding TF-IDF

At its core, TF-IDF is a numerical statistic that reflects how important a word or phrase is to a document within a collection or corpus. It is the product of two metrics:

1. Term Frequency (TF): This measures how frequently a term appears in a document. The raw count is usually normalized to prevent bias towards longer documents.
2. Inverse Document Frequency (IDF): This measures the informativeness of a term. Terms that are very common across the corpus (e.g. "the", "and", "a") receive a lower weight, while rare terms are considered more informative and receive a higher weight.

The intuition behind TF-IDF is that the best key phrases for a document will be those that appear frequently in that document, but infrequently in the corpus as a whole. This helps surface phrases that are uniquely relevant to the main topics of that specific document.

For example, consider a document about machine learning. The phrase "gradient descent" will likely have a high TF-IDF score, since it is a phrase that appears often when discussing machine learning algorithms, but relatively rarely across a broad corpus of documents on many topics.

In contrast, a phrase like "the algorithm" might appear frequently in the machine learning document, but also very commonly across the full corpus, giving it a lower TF-IDF score and making it less useful as a key phrase.

By ranking all candidate phrases in a document by their TF-IDF score, we can automatically extract those that are most representative of the document‘s contents. Next, let‘s see how to implement this in Python.

## Extracting Key Phrases with Python

We‘ll now walk through the steps to extract key phrases from a set of documents using TF-IDF with Python‘s scikit-learn library. The high-level process is:

1. Prepare a dataset of text documents
2. Preprocess and clean the text
3. Generate n-grams as candidate phrases
4. Calculate TF-IDF weights for each n-gram
5. Rank n-grams by TF-IDF weight to get top key phrases

Let‘s go through each step in detail.

### 1. Preparing the Dataset

The first step is to obtain a collection of documents that you want to extract key phrases from. These could be articles, reports, blog posts, or any other text data. For this example, we‘ll use the Hulth-2003 dataset which contains 2000 scientific abstracts from the Inspec database with manually assigned key phrases.

You can download the dataset from the [author‘s website](https://www.aclweb.org/anthology/W03-1028.pdf). We‘ll assume the files are stored locally in a folder called `data`.

We can load the documents and key phrases into a Pandas DataFrame:

```
import pandas as pd

docs = []
labels = []

for file_name in os.listdir("data/Hulth2003/Test/"):
    if file_name.endswith(".abstr"):
        with open(f"data/Hulth2003/Test/{file_name}", "r") as file:
            docs.append(file.read())
        with open(f"data/Hulth2003/Test/{file_name[:-6]}.contr", "r") as file:
            labels.append(file.read().split("\n"))

df = pd.DataFrame({"text": docs, "key_phrases": labels})
df.head()
```

This gives us a DataFrame with one row per document, a `text` column with the abstract text, and a `key_phrases` column with a list of the manually assigned key phrases.

### 2. Text Preprocessing

With our data loaded, the next step is to clean and preprocess the text to remove noise and standardize formatting. This typically includes:

- Removing numbers, punctuation, and special characters
- Converting to lowercase
- Removing stop words
- Lemmatizing or stemming words

Here‘s an example function to preprocess a string of text:

```
import re
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

stop_words = set(stopwords.words("english"))
lemmatizer = WordNetLemmatizer()

def preprocess(text):
    # Remove special characters and lowercase
    text = re.sub(r"[^a-zA-Z0-9]", " ", text.lower())

    # Tokenize
    tokens = nltk.word_tokenize(text)

    # Remove stopwords and lemmatize
    tokens = [lemmatizer.lemmatize(token) for token in tokens if token not in stop_words]

    return " ".join(tokens)
```

We can apply this function to the `text` column of our DataFrame:

```
df["cleaned_text"] = df["text"].apply(preprocess)
```

### 3. Generating Candidate Phrases

With our text cleaned, the next step is to generate candidate phrases to consider. The simplest approach is to use a sliding window to extract all n-grams up to a certain length.

For example, if we set the n-gram range to (1, 3), we‘ll consider all unigrams, bigrams and trigrams in each document as potential key phrases. In practice, phrases longer than 3 words are relatively rare, so this is often a good range to use.

We‘ll use scikit-learn‘s `TfidfVectorizer` to handle both generating n-grams and calculating the TF-IDF weights in the next step. To generate n-grams, we set the `ngram_range` parameter:

```
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(ngram_range=(1,3))
X = vectorizer.fit_transform(df["cleaned_text"])
```

The `fit_transform` method fits the vectorizer on our corpus and returns the TF-IDF weighted document-term matrix. Each row corresponds to a document, and each column corresponds to an n-gram.

### 4. Calculating TF-IDF Weights

With the vectorizer fit on our data, the TF-IDF weights have already been calculated. The `X` matrix from the previous step contains the TF-IDF score for each n-gram in each document.

However, it can be useful to understand a bit more about how these scores are calculated. The exact formulas used by scikit-learn are:

$$ \text{TF}(t,d) = \frac{f_{t,d}}{\sum_{t‘ \in d} f_{t‘,d}} $$

$$ \text{IDF}(t) = \log \left( \frac{1 + n}{1 + \text{df}_t} \right) + 1 $$

$$ \text{TF-IDF}(t,d) = \text{TF}(t,d) \times \text{IDF}(t) $$

Where:

- $f_{t,d}$ is the raw count of term $t$ in document $d$
- $\text{df}_t$ is the number of documents that contain term $t$
- $n$ is the total number of documents

The TF is normalized by the total number of terms in the document to account for different document lengths. The IDF includes a smoothing factor to avoid divide-by-zero issues.

Some key parameters of the `TfidfVectorizer` to be aware of are:

- `max_df`: This sets the maximum document frequency for a term to be included. The default is 1.0, meaning terms can appear in 100% of documents. A lower value like 0.7 would exclude any terms that appear in more than 70% of documents.
- `min_df`: This sets the minimum number of documents a term must appear in to be included. The default is 1, meaning a term must appear in at least 1 document. Increasing this can help filter out very rare terms.

It‘s often worth experimenting with different values for these parameters to find what works best for a given dataset and use case.

### 5. Extracting Top Key Phrases

With our n-grams generated and TF-IDF weights calculated, we‘re finally ready to extract the top key phrases for each document.

First, let‘s get the mapping from column index to n-gram string. The vectorizer‘s `get_feature_names()` method returns the n-grams in the same order as the columns of the TF-IDF matrix:

```
feature_names = vectorizer.get_feature_names()
```

Now we can iterate through each row of the matrix, sort the n-grams by their TF-IDF score, and take the top N as the key phrases for that document:

```
def extract_topn_from_vector(feature_names, sorted_items, topn=10):
    return [feature_names[idx] for idx, score in sorted_items[:topn]]

top_n = 10
key_phrases = []

for i, row in enumerate(X):
    row = row.toarray()[0]
    sorted_items = sorted([(idx, val) for idx, val in enumerate(row)], key=lambda x: x[1], reverse=True)
    key_phrases.append(extract_topn_from_vector(feature_names, sorted_items, top_n))

df["tfidf_key_phrases"] = key_phrases
```

We can now compare the extracted key phrases to the ground truth manual labels:

```
df[["tfidf_key_phrases", "key_phrases"]].head()
```

## Evaluating Performance

To quantitatively evaluate the performance of our TF-IDF key phrase extraction, we can use the mean average precision at K (MAP@K) metric. This measures the average precision over all documents of considering the top K extracted key phrases.

First, let‘s write a function to calculate the precision at K for a single document:

```
def apk(actual, predicted, k=10):
    if len(predicted) > k:
        predicted = predicted[:k]

    score = 0.0
    num_hits = 0.0

    for i, p in enumerate(predicted):
        if p in actual and p not in predicted[:i]:
            num_hits += 1.0
            score += num_hits / (i + 1.0)

    return score / min(len(actual), k)
```

And then we can calculate the MAP@K by averaging over all documents:

```
def mapk(actual, predicted, k=10):
    return np.mean([apk(a, p, k) for a, p in zip(actual, predicted)])

k = 10
map_k = mapk(df["key_phrases"], df["tfidf_key_phrases"], k)
print(f"Mean Average Precision at {k}: {map_k:.3f}")
```

On this dataset, the TF-IDF method achieves a MAP@10 of around 0.32. While this is a decent result, it shows there is certainly room for improvement. More sophisticated methods that leverage word embeddings or transformer language models can often achieve better performance.

However, TF-IDF remains a strong baseline and a great starting point given its simplicity and efficiency. It‘s often worth trying as a first pass before moving on to more complex approaches.

## Tips and Best Practices

To wrap up, here are a few tips and best practices to keep in mind when using TF-IDF for key phrase extraction:

- Experiment with different n-gram ranges, `max_df`, and `min_df` parameters to find what works best for your specific dataset and goals
- Use document frequency thresholds to filter out both very common and very rare terms that are less likely to be informative
- Be aware that TF-IDF can struggle with extracting key phrases containing out-of-vocabulary or infrequent terms
- Consider combining TF-IDF with other methods like part-of-speech tagging or named entity recognition as a way to generate higher quality candidate phrases
- Manually review the extracted key phrases for a sample of documents to assess quality and identify potential areas for improvement

## Conclusion

In this guide, we took an in-depth look at key phrase extraction using the TF-IDF algorithm. We covered:

- What TF-IDF is and the intuition behind why it works for identifying important phrases
- How to implement TF-IDF in Python using scikit-learn‘s `TfidfVectorizer`
- The importance of text preprocessing and n-gram generation in the key phrase extraction pipeline
- How to evaluate performance using the mean average precision metric
- Tips and best practices for getting the most out of TF-IDF

For further reading, I recommend checking out the [scikit-learn TF-IDF documentation](https://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting) and this [survey paper on key phrase extraction](https://arxiv.org/abs/1905.05044) for an overview of other techniques.

I hope this guide has given you a solid foundation for working with TF-IDF and extracting key phrases in Python. Feel free to reach out with any questions!

---

Source: [A Comprehensive Guide to Key Phrase Extraction with TF\-IDF in Python](https://33rdsquare.com/how-to-extract-key-phrases-using-tfidf-with-python/)