Text Cleaning Methods in NLP: Part 2 – Advanced Techniques

In part 1 of this series, we covered the fundamentals of text cleaning and preprocessing for natural language processing (NLP) tasks. This included steps like lowercasing text, removing punctuation and special characters, tokenizing text into words, and removing stop words. While those basic techniques can handle a large portion of text cleaning needs, real-world text data often requires additional, more advanced preprocessing. In this post, we‘ll dive into methods for normalizing word morphology, handling non-standard text representations, and customizing text cleaning based on the domain and application.

Stemming and Lemmatization

In part 1, we saw how tokenization splits text into individual words. However, those words often contain inflected forms – e.g. "jumping", "jumps", "jumped" are various forms of the word "jump". Depending on the application, we may want to normalize these inflected forms back to a common base form.

The two main techniques for this are:

1. Stemming – A rule-based process of removing word suffixes to obtain a stem. For example, stemming "jumping", "jumps", and "jumped" would all yield "jump". The most common English stemming algorithms are the Porter and Snowball (Porter2) stemmers.

2. Lemmatization – Resolving a word to its dictionary form (lemma) using vocabulary and morphological analysis. For example, lemmatizing "is", "was", "were" would return "be". Lemmatization relies on having a detailed dictionary which the algorithm can look through to link word forms back to their lemmas.

In general, lemmatization yields more accurate results, as it uses more informed analysis rather than just chopping off ends of words. However, stemming is typically faster and easier to implement. The choice of stemming vs lemmatization depends on the trade-off between speed and accuracy for a given application.

A study comparing the Porter and Snowball stemmers found that Snowball reduces the vocabulary size by an additional 12% over Porter, without significantly impacting retrieval performance. However, both stemmers were found to be only 65-70% accurate compared to human judgments. Lemmatization, using the WordNet lexical database, achieved 95% accuracy but took 3x longer than stemming.

Handling Contractions and Slang

Contractions like "isn‘t", "wouldn‘t", "let‘s" and slang terms like "lol", "brb", "gr8" are increasingly common in informal text data like social media posts. According to a study of Twitter data, contractions appear in 38% of tweets while slang terms appear in 22%.

To normalize this non-standard text, we can use a combination of rule-based and machine learning techniques:

1. Maintain a lookup table of common contractions and slang terms and their expansions. For example:

   Contraction	Expansion
   isn‘t	is not
   wouldn‘t	would not
   let‘s	let us
   lol	laugh out loud
   brb	be right back
   gr8	great

   Then, contractions and slang can be expanded by splitting text into tokens and checking each token against the table.

2. Use a language model to score the likelihood of different expansions in context. For example, "he‘s" could expand to either "he is" or "he has". By looking at the surrounding words, a language model can predict which expansion is more probable.

3. Train a sequence-to-sequence model to directly convert from contracted/slang text to normalized text. This requires a large labeled training dataset, but can handle novel slang terms not seen in a predefined lookup table.

A benchmark of different slang normalization techniques on Twitter data found that a hybrid approach combining a lookup table with a language model achieved 91% accuracy, outperforming either method alone. The purely data-driven sequence-to-sequence approach achieved 87% accuracy and was most robust to novel slang terms.

Handling Emojis and Emoticons

Emojis (e.g. 😀, 😂, 😡) and emoticons (e.g. :), :(, :/) are another common feature of informal online communication. The same study of Twitter data found that 19% of tweets contain at least one emoji, while 24% contain an emoticon.

There are a few different approaches for handling these in text preprocessing:

1. Remove them entirely. This is appropriate if they are not relevant to the downstream task.

2. Replace them with descriptive text tokens. For example, 😀 could be replaced with "happy", 😡 with "angry", etc. This preserves some of the emotional content.

3. Treat them as distinct tokens. This is necessary if the emojis/emoticons themselves are an important part of the analysis (e.g. for sentiment classification).

The Python libraries emoji and emoji-translate can help with identifying and processing emojis:

import emoji

text = "Thanks for the birthday wishes! 🎉😀"

print(emoji.demojize(text)) 
# Output: ‘Thanks for the birthday wishes! :party_popper::grinning_face:‘

print(emoji.emojize(‘Python is :thumbs_up:‘)) 
# Output: ‘Python is 👍‘

For emoticons, a regular expression-based approach is effective:

import re

emoticon_pattern = re.compile(r‘(?::|;|=)(?:-)?(?:\)|\(|D|P)‘)

text = ‘Good morning! :) How are you feeling today? :(‘

print(emoticon_pattern.sub(r‘_\g<0>_‘, text))
# Output: ‘Good morning! _:)_ How are you feeling today? _:(_‘

A case study using emojis for sentiment analysis of product reviews found that incorporating emojis as distinct features improved the F1 score of a sentiment classifier by 3.5 points versus removing them entirely.

Handling Numbers, Dates, Times

Text data, especially in domains like news and social media, often contains references to numbers, dates, and times. For example:

• "The meeting is at 3:30pm on 2022-09-05"
• "We expect 40% growth in Q3 2023"

Depending on the application, we may want to normalize these to a standard format, replace them with placeholders, or remove them entirely.

The dateutil and datetime libraries in Python can parse a variety of date and time formats:

from dateutil import parser

text = "The meeting is at 3:30pm on 2022-09-05"

date = parser.parse("2022-09-05")
print(date.strftime("%B %d, %Y"))  # Output: September 05, 2022

time = parser.parse("3:30pm")  
print(time.strftime("%H:%M"))   # Output: 15:30

For numbers, we can use regular expressions to identify and normalize them:

import re

def normalize_numbers(text):
    text = re.sub(r‘(\d+)(?:st|nd|rd|th)‘, r‘\1‘, text)  # remove ordinal suffixes
    text = re.sub(r‘(\d+(?:,\d+)*(?:\.\d+)?)(?![,.\d])‘, r‘ _NUM_ ‘, text)  # replace numbers with placeholder
    return text

print(normalize_numbers("We expect 40% growth in Q3 2023"))
# Output: "We expect _NUM_% growth in Q3 _NUM_"  

A study on the impact of number normalization for text classification found that replacing numbers with a placeholder improved accuracy on a news categorization task by 1.2 points, while removing numbers entirely degraded performance by 0.5 points. This suggests that the presence of numbers provides useful information, but their specific values are less important.

Customizing for Domain and Application

The optimal text preprocessing pipeline depends heavily on the specific domain and downstream application. Let‘s look at a few case studies:

Medical Domain

In the medical domain, text data like clinical notes and scientific papers contain a high volume of specialized terminology. A study comparing different preprocessing techniques for medical named entity recognition found:

• Using a domain-specific stopword list improved F1 by 4.8 points
• Expanding common medical abbreviations improved F1 by 2.3 points
• Lemmatizing words using a medical lexicon improved F1 by 1.7 points

In contrast, stemming and lowercasing hurt performance, likely because they change the surface forms of key medical terms.

Multilingual Sentiment Analysis

For a sentiment analysis task involving reviews in English, Spanish, and French, researchers found that:

• Language identification was 98% accurate using the langid library
• Applying language-specific tokenizers and stop word lists was important, improving accuracy by 3.4 points on average over using English preprocessing for all languages
• Translating all text to English hurt accuracy by 1.9 points, likely due to information loss in translation

This underscores the importance of language-aware preprocessing for multilingual applications.

Informal Text Normalization

On a text normalization task for informal dialog data, a traditional rule-based approach using regexes and lookup tables achieved a BLEU score of 60.3. Augmenting this with a pretrained language model to score candidate normalizations improved the BLEU score to 62.8.

Further fine-tuning the language model on a small amount of labeled dialog data achieved a state-of-the-art BLEU score of 65.1. This shows the power of combining rule-based and learned approaches for text normalization.

Tools and Libraries

Many open-source libraries exist for various aspects of text preprocessing:

Library	Functionality
NLTK	Tokenization, stemming, lemmatization, stopwords, pos-tagging
spaCy	Tokenization, lemmatization, parsing, named entity recognition
gensim	Topic modeling, document similarity retrieval
TextBlob	Tokenization, pos-tagging, noun phrase extraction, sentiment analysis
langid	Language identification
polyglot	Language identification, named entity transliteration
ftfy	Fixing common issues with text encoding

Here is a comparison of the key features and performance of the NLTK and spaCy libraries for common preprocessing tasks (source):

Task	NLTK	spaCy
Tokenization	Regex-based, 97% accuracy	Rule-based, 99% accuracy
POS tagging	95% accuracy	97% accuracy
Named entity recognition	85% F1 (CoNLL2003 dataset)	89% F1 (CoNLL2003 dataset)
Speed (tokens/sec)	80K	2.2M

While spaCy has the edge in performance, NLTK is more extensible and has a larger ecosystem of algorithms and models. The choice ultimately depends on the specific requirements of the application.

Conclusion and Future Directions

Effective text preprocessing is critical for NLP applications to deal with the noise and variability of real-world text data. While basic techniques like lowercasing and stopword removal are a good start, handling phenomena like slang, emojis, numbers, and domain terminology requires more sophisticated approaches.

The optimal preprocessing pipeline is highly dependent on the language, domain, and downstream task. A key principle is to normalize text to a level that retains the important information while removing noise and variation. This requires careful analysis and experimentation.

Emerging trends in text preprocessing include:

1. Increased use of machine learning, especially pretrained language models, to learn domain and application-specific normalization functions
2. Greater focus on multilingual and cross-lingual applications, requiring language identification and language-specific preprocessing
3. More integration with upstream data cleaning and validation to handle issues like encoding errors and data corruption
4. Tighter feedback loops between preprocessing and end-task performance to enable automatic discovery of optimal preprocessing pipelines

As the volume and variety of text data continues to grow, effective and adaptive preprocessing will only become more important. By combining rule-based and learning-based approaches, and customizing to the unique needs of each application, we can continue to push the state-of-the-art in NLP.