A Friendly Guide to NLP: Bag of Words with Python Example

If you‘re getting started with Natural Language Processing (NLP), one of the first techniques you‘re likely to come across is the Bag of Words model. Bag of Words is a simple but powerful way to represent text data numerically so it can be used for machine learning tasks like sentiment analysis, document classification, topic labeling, and more.

In this guide, we‘ll break down exactly what the Bag of Words model is, why it‘s useful, and how you can easily implement it yourself in Python. We‘ll walk through an example using the classic 20 Newsgroups dataset and scikit-learn so you can see Bag of Words in action and follow along step-by-step. Let‘s dive in!

What is Bag of Words?

At its core, Bag of Words (BoW) is a way to take text, which is readable by humans but not by machines, and convert it to a numeric vector representation that machine learning algorithms can understand and work with. It‘s called a "bag" of words because any information about the order or structure of words in the document is discarded. The model is only concerned with whether known words occur in the document, not where they occur in the document.

Here‘s how the process works at a high level:

1. Tokenization: The text is tokenized into individual words. So a string like "Hello world!" becomes ["Hello", "world", "!"].

2. Vocabulary Building: A vocabulary of unique words is built from the tokenized text. This is essentially the list of all different words that occur across the corpus of documents.

3. Encoding: Finally, each document is encoded into a numeric vector where each number represents the count of how often that vocabulary word appeared in the document.

The resulting vectors will have a length equal to the vocabulary size, with one number for every unique word. If a word from the vocabulary doesn‘t appear in a given document, its value in the vector will be zero.

Let‘s look at a quick example to solidify the concept. Say we have two movie reviews:

Review 1: This movie was amazing!

Review 2: I hated this movie, it was terrible.

First we tokenize the text and build the vocabulary:

Vocabulary:
[‘this‘, ‘movie‘, ‘was‘, ‘amazing‘, ‘!‘, ‘I‘, ‘hated‘, ‘it‘, ‘terrible‘, ‘.‘]

Then we encode each review into a vector based on word counts:

Review 1: [1, 1, 1, 1, 1, 0, 0, 0, 0, 0] Review 2: [1, 1, 1, 0, 0, 1, 1, 1, 1, 1]

So Review 1‘s vector has 1s in the positions of the words "this", "movie", "was", "amazing", and "!", while Review 2‘s vector has 1s for "this", "movie", "was", "I", "hated", "it", "terrible" and ".".

The beauty of this approach is that we‘ve taken unstructured text data and converted it to structured numeric vectors that machine learning models can learn from. Things like word frequencies and occurrences are quantified so the model can detect patterns.

Of course, this is a tiny toy example. In the real world, vocabularies get very large (tens or hundreds of thousands of words) and documents get long, leading to sparse, high-dimensional vector representations. But the core idea remains the same.

Implementing Bag of Words from scratch

Now that we understand the concept, let‘s see how to implement Bag of Words ourselves in Python. We‘ll use this simple corpus of documents as an example:

docs = [
 "The quick brown fox jumps over the lazy dog.",
 "A quick brown fox jumps over the lazy dog.",
 "The quick brown fox jumped over the lazy dog.",
 "The lazy dog is quick and brown.", 
]

Step 1 is tokenization. We‘ll use a simple regular expression to split on whitespace and punctuation:

import re

def tokenize(doc):
    return re.findall(r‘\w+‘, doc.lower())

tokenized_docs = [tokenize(doc) for doc in docs]
print(tokenized_docs)

[[‘the‘, ‘quick‘, ‘brown‘, ‘fox‘, ‘jumps‘, ‘over‘, ‘the‘, ‘lazy‘, ‘dog‘],
 [‘a‘, ‘quick‘, ‘brown‘, ‘fox‘, ‘jumps‘, ‘over‘, ‘the‘, ‘lazy‘, ‘dog‘],
 [‘the‘, ‘quick‘, ‘brown‘, ‘fox‘, ‘jumped‘, ‘over‘, ‘the‘, ‘lazy‘, ‘dog‘],
 [‘the‘, ‘lazy‘, ‘dog‘, ‘is‘, ‘quick‘, ‘and‘, ‘brown‘]]

Next, we build the vocabulary by finding all unique words across the tokenized documents:

from collections import Counter

def build_vocabulary(tokenized_docs):
    counter = Counter()
    for doc in tokenized_docs:
        counter.update(doc)
    return list(counter.keys())

vocabulary = build_vocabulary(tokenized_docs)
print(vocabulary)

[‘the‘, ‘quick‘, ‘brown‘, ‘fox‘, ‘jumps‘, ‘over‘, ‘lazy‘, 
 ‘dog‘, ‘a‘, ‘jumped‘, ‘is‘, ‘and‘]

Finally, we encode each document into a vector of word counts:

def encode_document(tokenized_doc, vocabulary):
    doc_vec = [0] * len(vocabulary)
    for token in tokenized_doc:
        if token in vocabulary:
            doc_vec[vocabulary.index(token)] += 1
    return doc_vec

encoded_docs = [encode_document(doc, vocabulary) for doc in tokenized_docs]
print(encoded_docs)

[[2, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0],
 [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0], 
 [2, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0],
 [1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1]]

Each vector has length 12, one for each word in the vocabulary. The values represent how many times that word occurred in the document. For example, we can see that the first document (index 0) has two occurrences of "the" (the first vocabulary word).

And there you have it – we‘ve implemented the core steps of the Bag of Words model! Of course, there are many additional preprocessing steps and optimizations we could add, but fundamentally Bag of Words encoding boils down to tokenizing, building a vocabulary, and counting word occurrences to create numeric vector representations.

Bag of Words with scikit-learn

Implementing bag of words from scratch is a good exercise to internalize how the encoding process works. However, for real-world applications, you‘ll want to leverage an optimized implementation from a library like scikit-learn.

Scikit-learn makes bag of words encoding extremely simple with its CountVectorizer class. CountVectorizer handles tokenization, vocabulary building, and encoding all in one. Let‘s quickly try it on the 20 newsgroups dataset, a collection of around 18,000 newsgroup posts across 20 different topics.

First, we load the data using scikit-learn‘s built-in newsgroups loader:

from sklearn.datasets import fetch_20newsgroups

newsgroups_data = fetch_20newsgroups(subset=‘train‘, 
                                     remove=(‘headers‘, ‘footers‘, ‘quotes‘))

This loads a subset of the data (the training set) and removes metadata like headers and footers that could bias the model. The result is a dictionary-like object where the data key contains the text of each newsgroup post.

Next, we instantiate a CountVectorizer and fit it to the data:

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(stop_words=‘english‘, max_features=1000)
X = vectorizer.fit_transform(newsgroups_data.data)

We set stop_words=‘english‘ to automatically remove common English words like "the", "and", "a" that occur very frequently but provide little informational content. We also set max_features=1000 to build the vocabulary from the 1000 most frequent words, ignoring very rare words.

After fitting, X is a sparse matrix of shape [n_documents, n_words] where entry X[i, j] is the count of word j in document i. We can inspect it to verify:

>>> X.shape
(11314, 1000)
>>> X[0].toarray()
array([[0, 0, 0, ..., 0, 0, 0]], dtype=int64)
>>> vectorizer.get_feature_names_out()[100:105]
array([‘access‘, ‘according‘, ‘account‘, ‘across‘, ‘act‘], ...)

We can see there are 11,314 documents encoded as 1000-dimensional vectors. The first document‘s vector is all zeros (not very interesting). The vocabulary words at indexes 100-105 include "access", "according", "account" and so on.

That‘s all there is to it! With just a few lines of scikit-learn code, we‘ve transformed over 10,000 raw text documents into numeric vectors ready for machine learning.

Of course, there are many ways to customize the vectorizer, like changing the tokenization pattern, setting max and min document frequencies, and using different vocabulary sizes. It‘s worth checking out the documentation to see all the knobs you can tune.

Beyond Bag of Words

As we‘ve seen, Bag of Words is a powerful way to numerically represent text data. However, it has some notable limitations:

• Word order is completely discarded. "Dog bites man" and "Man bites dog" will have the exact same representation.
• The vocabulary can get very large, leading to high dimensional, sparse vectors that become computationally expensive to work with.
• Very frequent words dominate and bias the encoding, while very rare but potentially informative words are often completely ignored.
• Semantic relationships between words are not captured at all. "Good" and "great" are treated as completely different features.

For these reasons, bag of words is just one of many text vectorization techniques. More advanced methods like TF-IDF weighting, word embeddings, and neural network-based approaches can address some of bag of words‘ shortcomings.

TF-IDF (term frequency-inverse document frequency) weighting is a next step from vanilla bag of words that weights words proportionally to how often they appear in a document, but inversely to their frequency across all documents. This helps surface more informative words while downweighting very common words.

Word embeddings like word2vec and GloVe learn to map words to dense vector representations such that semantically similar words have similar vectors. This allows embedding models to capture analogies and relationships between words.

Finally, modern NLP is increasingly relying on large pre-trained transformer models like BERT, GPT, RoBERTa etc. These models use complex neural network architectures to build rich representations of text that achieve state of the art performance on a wide variety of language tasks.

So while bag of words is a great starting point, it‘s just the beginning of a wide world of NLP vectorization techniques!

Conclusion

We‘ve taken an in-depth look at the bag of words model and seen how it can be used to transform raw text into a numeric representation suitable for machine learning.

We walked through how to implement bag of words from scratch in Python, then saw how scikit-learn‘s CountVectorizer makes the process much simpler.

Some key takeaways:

• Bag of words discards word order and structure – it‘s just concerned with what words occur and how often
• The vocabulary of unique words across documents can get very large, leading to high-dimensional, sparse vectors
• Bag of words can be a good baseline for simple applications, but more advanced techniques are needed for state of the art performance

I hope this guide has helped you understand the fundamentals of this important NLP technique. For next steps, I‘d encourage you to try bag of words on your own datasets, explore the scikit-learn CountVectorizer in more depth, and start learning about more advanced methods like TF-IDF and word embeddings.

Thanks for reading, and happy NLP-ing!