output: [‘This‘, ‘is‘, ‘a‘, ‘sample‘, ‘sentence‘, ‘.‘, ‘Here‘, ‘is‘, ‘another‘, ‘one‘, ‘!‘]

As the volume of unstructured text data continues to grow exponentially, extracting insights and meaning from this data is becoming increasingly important. Natural language processing (NLP) techniques allow us to analyze and understand human language at scale. However, raw text data is messy and unstructured. Before applying advanced NLP algorithms, it‘s critical to clean and preprocess the text data. Proper preprocessing can significantly improve the quality of results from NLP models.

In this article, we‘ll dive into 10 fundamental text preprocessing techniques in Python that every NLP practitioner should know. Having a solid grasp of these will enable you to effectively work with textual data and set you up for success with more advanced NLP tasks. Let‘s get started!

1. Text Cleaning

The first step is to clean the raw text and remove any unwanted characters or artifacts. Common cleaning steps include:

• Removing HTML tags or other markup
• Eliminating extra whitespaces, tabs, and newline characters
• Removing special characters and punctuation
• Converting all text to the same case (lowercase or uppercase)

Python provides built-in string methods that make cleaning text easy. For example:

text = "  This is a messy \t string! "
cleaned_text = text.strip().lower().replace("\t", "")
print(cleaned_text)
# output: this is a messy string!

The strip() method removes leading/trailing whitespace, lower() converts to lowercase, and replace() substitutes the tab \t with an empty string, effectively deleting it. Chaining these methods together allows us to clean the text in a single line of code.

2. Tokenization

Tokenization refers to splitting text into smaller units called tokens. Typically, text is tokenized into words or sentences. Tokenizing text is a prerequisite for many NLP tasks.

Python‘s NLTK library provides convenient methods for tokenizing text:

from nltk.tokenize import word_tokenize, sent_tokenize
text = "This is a sample sentence. Here is another one!"
words = word_tokenize(text)
print(words)

sentences = sent_tokenize(text)
print(sentences)  

3. Stop Word Removal

Stop words are commonly occurring words that typically add little meaning, such as "a", "an", "the", "is", etc. Removing stop words can help focus on the important words and significantly reduce the dataset size.

NLTK conveniently provides a list of stop words:

from nltk.corpus import stopwords
text = "This is a sample sentence with some stop words."
print(text)
stop_words = set(stopwords.words(‘english‘)) 
words = word_tokenize(text)
filtered_words = [word for word in words if word.lower() not in stop_words]
print(filtered_words)

4. Stemming

Stemming refers to normalizing words to their base or root form. For example, the words "running", "runs", and "ran" would all be reduced to the stem "run". Stemming helps group together different variations of the same word.

The Porter stemming algorithm is a popular choice for stemming English words. NLTK provides an implementation:

from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
words = ["running", "runs", "ran"]
stemmed_words = [stemmer.stem(word) for word in words]
print(stemmed_words)

Note that the stemmed words may not always be valid English words. For example, "studies" gets stemmed to "studi". Stemming is often sufficient and faster for many applications, but occasionally the full words are needed.

5. Lemmatization

Lemmatization is a more advanced form of stemming that reduces words to their base dictionary form, called a lemma. Unlike stemming, lemmatization considers the context and part of speech of a word to determine its lemma. The resulting lemma is always a valid English word.

The WordNet lemmatizer in NLTK is a popular choice:

 
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("better", pos="a"))

print(lemmatizer.lemmatize("better", pos="v"))

Here, the pos parameter specifies the part of speech, either "a" for adjective or "v" for verb. Lemmatizing "better" as an adjective returns "good", while lemmatizing it as a verb returns "better". Lemmatization is more accurate than stemming but also slower.

6. Part-of-Speech Tagging

Part-of-speech (POS) tagging assigns a grammatical category to each word in a sentence, such as noun, verb, adjective, etc. POS tags can provide useful information about a word‘s meaning and context. Some NLP algorithms use POS tags as additional features during training.

NLTK provides a function to get the POS tag for each word:

from nltk import pos_tag
text = "They refuse to permit us to obtain the refuse permit"
tagged_words = pos_tag(word_tokenize(text))
print(tagged_words)

The tags follow the Penn Treebank tagset. For example, NNP represents a singular proper noun, VB a verb in base form, and JJ an adjective.

7. Named Entity Recognition

Named entity recognition (NER) locates and classifies named entities in text into predefined categories, such as person names, organizations, locations, time expressions, monetary values, etc. NER can help extract structured information from unstructured text.

Popular Python libraries like NLTK and spaCy provide interfaces for performing NER:

import spacy
nlp = spacy.load("en_core_web_sm")
text = "Apple is looking to buy U.K. startup for $1 billion, Bloomberg reports."
doc = nlp(text)
for ent in doc.ents:
print(ent.text, ent.label_)

Here, spaCy identifies "Apple" and "Bloomberg" as organizations, "U.K." as a geopolitical entity, and "$1 billion" as money. Training custom NER models on domain-specific datasets can further improve performance.

8. Removing Frequent/Rare Words

In some cases, it can be beneficial to remove words that appear too frequently or too rarely. Frequent words are often less informative, while rare words can be noise or outliers. Removing these words can improve computational efficiency and predictive performance.

A common approach is to calculate each word‘s frequency and filter out words above or below a certain count threshold:

from collections import Counter
text = "This is a sample text with several words " 
"with some words repeated many times is is is"
word_counts = Counter(word_tokenize(text))
print(word_counts)  


min_count = 2
filtered_words = [word for word in word_counts if word_counts[word] >= min_count]
print(filtered_words)

Here, only words occurring at least twice are preserved. The optimal frequency thresholds will depend on the specific application and dataset.

9. Handling Contractions

Contractions are shortened versions of words or syllables, such as "isn‘t" for "is not" or "she‘ll" for "she will". Contractions can make text preprocessing tricky since they often involve joining multiple words. A simple approach is to maintain a dictionary of contractions and their expansions:

contractions = { 
  "isn‘t": "is not",
  "she‘ll": "she will",
  "couldn‘t": "could not"
}
def expand_contractions(text, contractions):
for contraction, expansion in contractions.items():
text = text.replace(contraction, expansion)
return text
text = "She isn‘t feeling well, but she‘ll be alright."

print(expand_contractions(text, contractions))

The expand_contractions function replaces each contraction in the text with its expansion. Of course, this simple approach won‘t handle all possible contractions. For a more comprehensive solution, consider using a library like contractions.

10. Regular Expressions

Regular expressions are a powerful tool for advanced pattern matching and manipulation of strings. They allow you to concisely search, extract, and modify text using special sequences of characters.

For example, let‘s extract all email addresses from a text:

import re
text = "For more information, contact [email protected] or [email protected]"
emailpattern = r‘\b[A-Za-z0-9.%+-]+@[A-Za-z0-9.-]+.[A-Z|a-z]{2,}\b‘
emails = re.findall(email_pattern, text)
print(emails) 

The regular expression email_pattern matches email addresses based on a generic pattern. re.findall finds all matching occurrences in the text. Regular expressions are a deep topic, but understanding the basics can be immensely useful for processing text.

Conclusion

In this article, we covered 10 essential techniques for preprocessing text data in Python, including cleaning, tokenization, stemming, lemmatization, stop word removal, POS tagging, NER, frequency filtering, expanding contractions, and regular expressions. However, we‘ve only scratched the surface of each technique and the available Python NLP libraries.

There are many other powerful libraries and packages worth exploring, such as spaCy for efficient NLP pipelines, gensim for topic modeling, and TextBlob for simplified text processing. The field of NLP is rapidly evolving, with new state-of-the-art models and frameworks like Transformers and HuggingFace accelerating progress.

Remember, the specific preprocessing steps and libraries you choose will depend on your particular application and requirements. It‘s important to experiment, iterate, and test different approaches to see what works best for your problem.

I hope this article has provided you with a solid foundation and jumping-off point for working with text data in Python. With practice and experience, you‘ll be well on your way to leveraging the full potential of natural language processing. The ability to extract meaning and insights from the vast amounts of unstructured text data is an incredibly valuable skill in today‘s data-driven world.

Best of luck on your NLP journey! Feel free to connect with me on LinkedIn or check out my other NLP articles on Analytics Vidhya.