Must Known Techniques for text preprocessing in NLP

Must-Know Text Preprocessing Techniques for NLP
Natural Language Processing, or NLP for short, is a fascinating field of artificial intelligence that focuses on enabling computers to understand, interpret, and generate human language. At the heart of many NLP applications and models lies text preprocessing – the crucial step of cleaning and normalizing text data to improve the quality of results.
In this comprehensive guide, we‘ll dive deep into the most important text preprocessing techniques that every NLP practitioner should know. Whether you‘re working on text classification, machine translation, sentiment analysis, or any other NLP task, understanding and applying these techniques effectively will significantly boost your model‘s performance. So let‘s get started!

Why is Text Preprocessing Important in NLP?

Raw text data, especially from online sources, can be incredibly noisy and unstructured. It often contains irrelevant or redundant information like HTML tags, extra whitespaces, punctuation, special characters, and more. This messy data is difficult for NLP models to make sense of.
Text preprocessing aims to clean up the noise and normalize the data into a more consistent format. This helps reduce the vocabulary size, distill text to its most meaningful parts, and allow models to focus only on the essential information for the task at hand. Models trained on properly preprocessed data tend to have better predictive power, faster training times, and require less memory.

Essential Text Preprocessing Steps

Let‘s now walk through the fundamental text preprocessing techniques, along with Python code samples, that you can apply to almost any text dataset.

1. Expanding Contractions

Contractions are shortened versions of words or syllables, like "don‘t" for "do not" or "you‘re" for "you are". They are very common in informal writing and speech. Expanding contractions helps standardize the text.

import re

def expand_contractions(text):
  contractions = { 
    "ain‘t": "are not",
    "aren‘t": "are not",
    "can‘t": "cannot",
    "can‘t‘ve": "cannot have",
    "could‘ve": "could have",
    "‘s": " is"
    }

  contractions_re = re.compile(‘(%s)‘ % ‘|‘.join(contractions.keys()))
  def replace(match):
    return contractions[match.group()] 

  return contractions_re.sub(replace, text)

# Example
text = "I ain‘t going there. Could‘ve been fun."  
print(expand_contractions(text))

Output:

I are not going there. Could have been fun.

2. Converting to Lowercase

Converting the entire text to lowercase helps treat words like "Hello", "HELLO" and "hello" the same. It reduces the vocabulary size and allows the model to learn more accurate representations.

text = "This is a sample sentence. LET‘S LOWERCASE IT!" 
lowercased = text.lower()
print(lowercased)

Output:

this is a sample sentence. let‘s lowercase it!

3. Removing Punctuation

Punctuation symbols like commas, periods, exclamation marks generally don‘t contribute to the main information in a text. Removing them helps declutter the data.

import re

def remove_punctuation(text):
  return re.sub(r‘[\.\?\!\,\:\;\-\=]‘, ‘‘, text)

text = "This is a sample sentence. It contains punctuation!" 
print(remove_punctuation(text))

Output:

This is a sample sentence It contains punctuation

4. Removing Numbers and Words Containing Numbers

Numbers, especially in combination with words, can greatly increase the vocabulary size without providing much useful information in most NLP tasks. It‘s often helpful to remove them.

import re

def remove_numbers(text):
  return re.sub(r‘\w*\d\w*‘, ‘‘, text)

text = "There are 7 days in a week. Humans have 2 eyes."
print(remove_numbers(text))  

Output:

There are  days in a week. Humans have  eyes.

5. Removing Stopwords

Stopwords are the most common words in a language like "the", "is", "in" etc. They don‘t carry much meaning and can be safely removed in most cases to reduce the data size.

from nltk.corpus import stopwords

stop_words = set(stopwords.words(‘english‘))

def remove_stopwords(text):
  return " ".join([word for word in text.split() if word not in stop_words])

text = "This is a sample sentence demonstrating stopword removal."  
print(remove_stopwords(text))

Output:

sample sentence demonstrating stopword removal.

You can also add your own domain- or task-specific stopwords:

my_stopwords = ["sample", "demonstrating"]
stop_words = stop_words.union(my_stopwords)

6. Text Normalization

Normalization involves converting words to a more uniform sequence to reduce redundancy. Some common examples are:

• Converting website URLs to a generic token like {link}
• Converting email IDs to {email}
• Converting numbers to {number}
• Converting currencies to {currency}
• Converting phone numbers to {phone}
• Removing accent marks
• Expanding abbreviations

Here‘s an example of normalizing URLs and email IDs:

import re

def normalize_text(text):
  text = re.sub(r‘(https|http)?:\/\/(\w|\.|\/|\?|\=|\&|\%)*\b‘, ‘{link}‘, text)  
  text = re.sub(r‘[\w\.-]+@[\w\.-]+‘, ‘{email}‘, text)
  return text

text = "Check out this link: https://www.example.com. Email me at [email protected]"
print(normalize_text(text))  

Output:

Check out this link: {link}. Email me at {email}  

7. Stemming and Lemmatization

Stemming and Lemmatization both generate the root form of the inflected words. The difference is that stemming operates on a single word without knowledge of the context, and therefore cannot discriminate between words that have different meanings depending on part of speech. Lemmatization uses a dictionary-based approach and considers the context to determine the correct lemma of each word.

Stemming:

from nltk.stem import PorterStemmer

def stem_text(text):
  stemmer = PorterStemmer()
  return " ".join([stemmer.stem(word) for word in text.split()])

text = "The boys are playing in the fully equipped playground."
print(stem_text(text))

Output:

the boy are play in the fulli equip playground.

Lemmatization:

from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet

def lemmatize_text(text):
  lemmatizer = WordNetLemmatizer()
  wordnet_map = {"N":wordnet.NOUN, "V":wordnet.VERB, "J":wordnet.ADJ, "R":wordnet.ADV}
  pos_tagged_text = nltk.pos_tag(text.split())
  return " ".join([lemmatizer.lemmatize(word, wordnet_map.get(pos[0], wordnet.NOUN)) for word, pos in pos_tagged_text])

text = "The boys are playing in the fully equipped playground."  
print(lemmatize_text(text))

Output:

The boy be play in the fully equipped playground

8. Removing Extra Whitespaces

Extra whitespaces between words can be removed to clean up the data using Python‘s built-in strip() method or regex substitutions.

import re

def remove_whitespace(text):
  return re.sub(‘ +‘, ‘ ‘, text)

text = "There   are  a  lot   of  extra   spaces   in   this   text."
print(remove_whitespace(text))

Output:

There are a lot of extra spaces in this text.

Additional Preprocessing Techniques

Depending on your specific NLP task and dataset, you may find these additional preprocessing techniques helpful:

• Tokenization: Split text into individual words, phrases or whole sentences.

  import nltk
  
  text = "This is a sample sentence for tokenization."
  
  # Word tokenization  
  word_tokens = nltk.word_tokenize(text)
  print(word_tokens)
  
  # Sentence tokenization
  sent_tokens = nltk.sent_tokenize(text)  
  print(sent_tokens)

  Output:

  [‘This‘, ‘is‘, ‘a‘, ‘sample‘, ‘sentence‘, ‘for‘, ‘tokenization‘, ‘.‘]
  [‘This is a sample sentence for tokenization.‘]  

• Removing HTML tags: If your text is scraped from webpages, it may contain a lot of HTML markup that needs to be removed.

  from bs4 import BeautifulSoup
  
  def strip_html_tags(text):
    soup = BeautifulSoup(text, "html.parser")
    stripped_text = soup.get_text()
    return stripped_text
  
  html_text = "<p><strong>This is a sample</strong> text with <em>HTML tags</em>.</p>"
  print(strip_html_tags(html_text))

  Output:

  This is a sample text with HTML tags.

• Spelling Correction: Misspelled words can increase the vocabulary size and hinder the model‘s ability to learn good representations. Basic spelling correction can be performed using libraries like TextBlob.

  from textblob import TextBlob  
  
  def correct_spelling(text):
    return TextBlob(text).correct()
  
  text = "This sentense has sum speling errors." 
  print(correct_spelling(text))

  Output:

  This sentence has some spelling errors.

• Handling Emojis and Emoticons: With the rise of social media data in NLP, handling emojis is becoming increasingly important. You can either remove them or convert them to corresponding words.

  import emoji
  
  def handle_emojis(text):
    return emoji.demojize(text)
  
  text = "This text has emojis! 😃 🎉" 
  print(handle_emojis(text))

  Output:

  This text has emojis! :grinning_face: :party_popper:

Preprocessing Considerations for Different NLP Tasks

The choice of preprocessing steps can vary based on the specific NLP task at hand. Here are a few general guidelines:

• Text Classification & Sentiment Analysis:

  • Remove stopwords, punctuation, numbers
  • Lowercase the text
  • Perform stemming or lemmatization
  • Normalize text (remove HTML tags, handle emojis, etc.)

• Machine Translation:

  • Tokenize text into individual sentences
  • Lowercase and remove punctuation
  • Normalize numbers, dates, currencies
  • Careful with stemming/lemmatization as they may change meaning

• Named Entity Recognition:

  • Avoid lowercasing as case can provide useful signals
  • Avoid removing numbers, punctuation as they can help identify entities
  • Avoid stemming/lemmatization

• Text Summarization:

  • Sentence tokenization is crucial
  • Avoid removing stopwords as they provide contextual information
  • Avoid aggressive stemming

Creating Reusable Preprocessing Functions

As you‘ve seen, applying each preprocessing step individually can be cumbersome. It‘s helpful to create reusable utility functions that can perform all the necessary preprocessing steps in one go. Here‘s an example:

import re
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

def preprocess_text(text):
  # Lowercase 
  text = text.lower()

  # Remove punctuation
  text = re.sub(r‘[\.\?\!\,\:\;\-\=\(\)]‘, ‘‘, text)

  # Remove numbers 
  text = re.sub(r‘\d+‘, ‘‘, text)

  # Remove stopwords
  stop_words = set(stopwords.words(‘english‘)) 
  text = " ".join([word for word in text.split() if word not in stop_words])

  # Lemmatize
  lemmatizer = WordNetLemmatizer()
  text = " ".join([lemmatizer.lemmatize(word) for word in text.split()])

  return text

You can now call this preprocess_text function on any text to apply all the steps at once:

text = "This is a sample sentence! It contains some stopwords, punctuations and a number 42." 
preprocessed_text = preprocess_text(text)
print(preprocessed_text)

Output:

sample sentence contains stopwords punctuation number

Other Useful NLP Libraries

While NLTK is a great library for performing various NLP tasks, here are a few other libraries you should check out:

• spaCy: Industrial-strength NLP library with a lot of preprocessing utilities and pre-trained models.
• gensim: Specialized library for topic modeling and document similarity retrieval.
• TextBlob: Provides a simple API for performing NLP tasks like pos-tagging, noun phrase extraction, sentiment analysis, and more.

Conclusion

We covered a lot of ground in this guide! We learned about the importance of text preprocessing in NLP, went through the essential preprocessing techniques with code examples, discussed additional techniques for handling complex data, and also saw how to create reusable preprocessing functions.

However, it‘s important to remember that not all techniques are applicable for all NLP tasks. The key is to experiment and find the right combination of preprocessing steps that work best for your specific use case.

Also, while text preprocessing is an essential first step, it alone doesn‘t guarantee good model performance. The quality and quantity of your data, choice of model architecture, and hyperparameter tuning all play critical roles in the success of your NLP application.

I hope this guide provided you with a solid foundation to preprocess your text data for NLP tasks. Happy coding!