The Ultimate Guide to Text Preprocessing for NLP: Part 3

Welcome back to our comprehensive guide on preparing textual data for natural language processing tasks. In the previous installments, we covered foundational NLP concepts and dove into common challenges encountered when working with raw, unstructured text data.

Now it‘s time to get our hands dirty and walk through the essential steps of cleaning and preprocessing text to transform it into a format suitable for machine learning models. By the end of this article, you‘ll have a solid understanding of key techniques and best practices, along with practical Python code snippets to implement them yourself.

But first, let‘s set the stage with some eye-opening statistics. Did you know:

• The average English speaker knows 20,000-35,000 words, but 90% of the average text contains fewer than 1000 unique words.
• Stopwords like "the", "and", "a" make up 20-30% of the total words in a typical document.
• There are over 170,000 words in current use in the English language. Efficient NLP models need to handle this immense vocabulary.
• Cleaning and preparing data is estimated to take up to 80% of a data scientist‘s time on any given project.

As you can see, text preprocessing is a crucial step that can‘t be overlooked. By filtering out noise and distilling text down to its most meaningful components, we allow models to focus on learning the important linguistic patterns and relationships.

Tokenization Techniques

The first step in cleaning text data is to break it down into smaller units called tokens. Tokenization can be performed at different levels of granularity:

Word Tokenization

This splits text into individual words using whitespace and punctuation as delimiters. For example, consider the sentence: "I love NLP, it‘s awesome!" Word tokenization would produce:
[‘I‘, ‘love‘, ‘NLP‘, ‘it‘, ‘s‘, ‘awesome‘]

While straightforward, word tokenization has some limitations. It doesn‘t handle contractions like "it‘s" well. And sometimes the punctuation is meaningful, like with "NLP,".

Sentence Tokenization

For some applications, it‘s better to keep sentences intact rather than breaking them down further. This is called sentence segmentation or sentence boundary detection.

Common approaches look for sentence-ending punctuation like periods, question marks, and exclamation points. However, not every period ends a sentence, which requires more sophisticated logic to handle correctly.

Regex-Based Tokenization

To overcome the limitations of the previous techniques, we can define custom tokenization rules using regular expressions. This allows handling complex cases like contractions, hyphenated words, numbers, and more.

For example, to split contractions while keeping punctuation, we could use:

import re

text = "I love NLP, it‘s awesome!"
print(re.findall(r"w+|s+|p{P}", text))

[‘I‘, ‘ ‘, ‘love‘, ‘ ‘, ‘NLP‘, ‘,‘, ‘ ‘, ‘it‘, "‘", ‘s‘, ‘ ‘, ‘awesome‘, ‘!‘]

The best tokenization approach depends on your end application. When in doubt, experiment!

Handling Noise and Unnecessary Content

With our text split into tokens, the next step is removing elements that are less relevant for learning, often called "noise". This includes:

Stopwords

As mentioned earlier, stopwords are extremely common words that appear in most documents, such as "the", "and", "is". They don‘t contribute much to the meaning and take up space in the input data.

Stopword removal can be done by comparing tokens against a preexisting list. NLTK provides stopwords for many languages:

from nltk.corpus import stopwords

stop_words = set(stopwords.words(‘english‘))
tokens = [t for t in tokens if not t in stop_words]

Punctuation

While punctuation can provide grammatical context, it‘s not always useful for the end NLP application. Punctuation can be filtered out using string translation:

import string

tokens = [t.translate(str.maketrans(‘‘, ‘‘, string.punctuation)) for t in tokens]

Numbers

Like punctuation, numbers may or may not be relevant depending on the application. If not, they can be removed with:

tokens = [t for t in tokens if not t.isdigit()]

Lowercasing

An easy way to reduce the vocabulary size is to lowercase all words. This way "The" and "the" are treated the same. Apply with:

tokens = [t.lower() for t in tokens]

Other Noise

Depending on the source of your text data, you may need to handle other types of noise and unnecessary content, such as:

• HTML tags and escape characters
• URLs
• Mentions, hashtags, emoji on social media
• Rare words
• Domain-specific stopwords (e.g. "patient" in medical records)

The possibilities for cleaning are almost endless and depend heavily on the application. A good rule of thumb is to always question if a piece of information is truly relevant before including it.

Normalization

Another important preprocessing step is text normalization – converting words to a canonical form to handle slight variations and reduce the overall vocabulary.

Stemming

Stemming refers to cutting off prefixes and suffixes to get to the root word. This can map related words like "running" and "run" to the same stem. The most widely used stemming algorithms are the Porter and Snowball (Porter2) stemmers:

from nltk.stem.snowball import SnowballStemmer

stemmer = SnowballStemmer(‘english‘)
tokens = [stemmer.stem(t) for t in tokens]

Lemmatization

Lemmatization is a more advanced technique that uses vocabulary and morphological analysis to reduce words to their dictionary form, called lemmas. It‘s more accurate than stemming but also slower.

Lemmatization requires knowing the part-of-speech (POS) tag for each word, since the lemma can vary by POS. For example, the lemma of "better" is "good" if used as an adjective, but "well" if used as an adverb.

from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet

wnl = WordNetLemmatizer()

def get_wordnet_pos(treebank_tag):
if treebank_tag.startswith(‘J‘):
return wordnet.ADJ
elif treebank_tag.startswith(‘V‘):
return wordnet.VERB
elif treebank_tag.startswith(‘N‘):
return wordnet.NOUN
elif treebank_tag.startswith(‘R‘):
return wordnet.ADV
else:
return ‘‘

lemmas = [] for word, pos in nltk.pos_tag(tokens):
wn_pos = get_wordnet_pos(pos)
if wn_pos == ‘‘:
lemmas.append(word)
else:
lemmas.append(wnl.lemmatize(word, pos=wn_pos))

POS Tagging

As seen in the previous section, part-of-speech (POS) tagging is the process of marking each word in a text with its grammatical category – noun, verb, adjective, etc. Knowing the POS of words is useful for:

• Word sense disambiguation
• Named entity recognition
• Improving word embeddings
• Dependency parsing
• Stopword removal
• Lemmatization
• and more

The most common approach is to train a statistical model to predict POS tags based on manually labeled training data. NLTK provides a pretrained tagger:

import nltk

text = nltk.word_tokenize("They refuse to permit us to obtain the refuse permit")
nltk.pos_tag(text)

[(‘They‘, ‘PRP‘), (‘refuse‘, ‘VBP‘), (‘to‘, ‘TO‘), (‘permit‘, ‘VB‘), (‘us‘, ‘PRP‘),
(‘to‘, ‘TO‘), (‘obtain‘, ‘VB‘), (‘the‘, ‘DT‘), (‘refuse‘, ‘NN‘), (‘permit‘, ‘NN‘)]

Visualizing Text Data

Before jumping into building NLP models, it‘s always a good idea to explore your preprocessed text data visually. Some common techniques are:

Word Clouds

Word clouds display the most frequent words in a body of text, with more frequent words appearing larger. They give a quick impression of the main topics. Several Python libraries exist for generating word clouds, such as Andreas Mueller‘s wordcloud.

Frequency Distributions

Plotting word frequencies can help identify imbalances and outliers in the data. We can use NLTK‘s FreqDist class:

from nltk import FreqDist

freq_dist = FreqDist(tokens)
freq_dist.plot(50, cumulative=False)

Going Further

We‘ve covered the core steps in cleaning text data, but there are many other techniques you may encounter:

• Spelling correction
• Removing frequent words
• Collocation extraction
• Detecting and removing near-duplicates
• Language detection
• Unicode normalization

More advanced linguistic techniques also exist, such as:

• Dependency parsing – extracting grammatical relationships between words
• Constituency parsing – breaking a text into sub-phrases
• Semantic role labeling – identifying the semantic arguments of predicates
• Coreference resolution – finding all mentions that refer to the same real-world entity

The right preprocessing steps will always depend on your end goal. The key is to remove noise that is unlikely to help, while retaining signal that is relevant to the task at hand.

Applications and benefits

Why put so much effort into text preprocessing? Because it has an outsized impact on the performance of downstream NLP applications. Some key benefits are:

• Improved accuracy – by focusing the model on the most meaningful information
• Reduced computational overhead – a smaller vocabulary means less memory and faster training
• Better generalization – models learn more useful patterns that apply across different examples
• Increased interpretability – standardizing the input leads to more coherent learned representations

Nearly every NLP application can benefit from appropriate preprocessing, including:

• Document classification and clustering
• Sentiment analysis
• Machine translation
• Text summarization
• Named entity recognition
• Chatbots and virtual assistants
• and many more

Text preprocessing is also highly relevant for processing non-English languages. While the core concepts are the same, the specific steps and tools may differ based on the linguistic characteristics of the language.

Conclusion

We‘ve covered a lot of ground in this guide to text preprocessing for NLP. While not always the most glamorous part of NLP projects, preprocessing is a critical step that can make or break your results.

The key takeaway is to always think carefully about what information is truly relevant, and don‘t be afraid to experiment with different approaches. Over time you‘ll develop an intuition for what works best in different scenarios.

Of course, we‘ve only scratched the surface of what‘s possible. As you dive deeper into NLP, you‘ll encounter more advanced techniques and considerations. But the concepts covered here should give you a solid foundation to build upon.

Now it‘s time to get out there and apply these techniques to your own NLP projects. Happy cleaning!