Convert to lowercase

ByJordan Brown

If you‘re working on any kind of Natural Language Processing (NLP) project, one of the first and most important steps is cleaning and preparing your text data. Raw text is messy – it contains irrelevant characters, inconsistent formatting and unnecessary words that can interfere with NLP models and lead to poor results.

This is where text cleaning comes in. Text cleaning is the process of transforming raw text into a more digestible format by removing irrelevant items, standardizing the data and reducing the vocabulary. By cleaning the text first, we end up with data that is more suitable for performing NLP tasks like sentiment analysis, text classification, Named Entity Recognition (NER) and so on.

Python‘s Natural Language Toolkit (NLTK) is the go-to library for processing textual data. It provides a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing and more. In this post, we‘ll focus on using NLTK for cleaning text and walk through code examples of key text cleaning techniques.

Getting Started with NLTK

Before we dive into text cleaning, let‘s set up NLTK. You can install it using pip:

pip install nltk

Then import the library in your Python script or notebook:

import nltk

NLTK also has some downloadable packages containing useful datasets like stopwords and trained models. You can download all the packages at once: 

nltk.download(‘all‘)

Or just download specific packages as needed, for example to get the stopwords package:

  
nltk.download(‘stopwords‘)

Now we‘re ready to start cleaning some text!

Text Cleaning Techniques with NLTK

There are many ways to clean and preprocess text data with NLTK depending on your specific needs. However, a typical text cleaning pipeline will involve most of the following steps:

  1. Convert text to lowercase
  2. Remove numbers
  3. Remove punctuation and special characters
  4. Tokenize text
  5. Remove stopwords
  6. Stem or lemmatize words

Let‘s go through each of these in detail with code examples. For demonstration purposes, we‘ll use this sample text throughout:

text = "This is a sample sentence. It contains punctuation, UPPERCASE LETTERS, and stop words."

1. Converting to Lowercase

A simple but important first step is to standardize the case of your text, usually by converting it to lowercase. This prevents the same words in different cases (e.g. "Hello" and "hello") from being treated as separate features.

In Python, you can convert a string to lowercase using the lower() method:

text = text.lower()
print(text)

Output:

this is a sample sentence. it contains punctuation, uppercase letters, and stop words.

2. Removing Numbers

Numbers usually don‘t contain useful information for NLP tasks so they can be removed. You can use regular expressions in Python to find and replace all numbers in the text with an empty string:

import re

text = re.sub(r‘\d+‘, ‘‘, text)
print(text)

Output:

this is a sample sentence. it contains punctuation, uppercase letters, and stop words.

3. Removing Punctuation and Special Characters

Similarly, punctuation marks and special characters like brackets, hyphens, percentage signs etc. do not add value for NLP in most cases. NLTK has a built-in list of punctuation marks which you can use to remove all punctuation from text: 

from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r‘\w+‘)
text = tokenizer.tokenize(text)
print(text)

Output:

[‘this‘, ‘is‘, ‘a‘, ‘sample‘, ‘sentence‘, ‘it‘, ‘contains‘, ‘punctuation‘, ‘uppercase‘, ‘letters‘, ‘and‘, ‘stop‘, ‘words‘]

The regular expression \w+ matches only word characters (letters, numbers if still present, and underscores), thereby removing punctuation and special characters.

4. Tokenization

Tokenization is the process of splitting text into smaller units called tokens, which usually correspond to words and punctuation. If you followed along the previous step, you‘ll notice the text has already been tokenized into a list of words.

If not, you can use NLTK‘s word_tokenize() function to tokenize a string into words:

from nltk.tokenize import word_tokenize

text = word_tokenize(text)

print(text)

Output: 

[‘this‘, ‘is‘, ‘a‘, ‘sample‘, ‘sentence‘, ‘it‘, ‘contains‘, ‘punctuation‘, ‘uppercase‘, ‘letters‘, ‘and‘, ‘stop‘, ‘words‘]

5. Removing Stopwords

Stopwords are common words that add little or no semantic value like "the", "is", "at", "which" etc. Removing them can reduce the size of the vocabulary without losing important information.

NLTK has a corpus of stopwords for various languages that you can use:

from nltk.corpus import stopwords  

stop_words = set(stopwords.words(‘english‘))


text = [word for word in text if word not in stop_words]
print(text)

Output:

  
[‘sample‘, ‘sentence‘, ‘contains‘, ‘punctuation‘, ‘uppercase‘, ‘letters‘, ‘stop‘, ‘words‘]

6. Stemming and Lemmatization

The final step is to normalize words to their base or dictionary form through stemming or lemmatization. Both reduce inflectional forms of words to a common base form, so that words like "run", "runs", "ran" and "running" are consolidated.

Stemming heuristically cuts off the ends of words to get the base form, called the stem. It‘s faster but cruder. For example:

from nltk.stem import PorterStemmer

stemmer = PorterStemmer()


text = [stemmer.stem(word) for word in text]
print(text)

Output:

[‘sampl‘, ‘sentenc‘, ‘contain‘, ‘punctuat‘, ‘uppercas‘, ‘letter‘, ‘stop‘, ‘word‘]

Notice how some stems like "sampl" and "punctuat" are not valid words.

Lemmatization, on the other hand, uses vocabulary and morphological analysis to return the dictionary form of a word, called the lemma. It‘s slower but more accurate:

  
from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()


text = [lemmatizer.lemmatize(word) for word in text]
print(text)

Output:

[‘sample‘, ‘sentence‘, ‘contains‘, ‘punctuation‘, ‘uppercase‘, ‘letter‘, ‘stop‘, ‘word‘]

The lemmatized words are all valid dictionary words, unlike some of the stems.

Putting it All Together

We can combine all the above steps into a single function to clean any text:

import re
from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

def clean_text(text):



text = text.lower()

# Remove numbers
text = re.sub(r‘\d+‘, ‘‘, text)

# Tokenize and remove punctuation
tokenizer = RegexpTokenizer(r‘\w+‘) 
tokens = tokenizer.tokenize(text)

# Remove stopwords
stop_words = set(stopwords.words(‘english‘)) 
tokens = [word for word in tokens if word not in stop_words]

# Lemmatize 
lemmatizer = WordNetLemmatizer()
tokens = [lemmatizer.lemmatize(word) for word in tokens]

return ‘ ‘.join(tokens)


Let‘s test it on our sample text:


  
text = "This is a sample sentence. It contains punctuation, UPPERCASE LETTERS, and stop words."
clean_text(text)



Output:


‘sample sentence contains punctuation uppercase letter stop word‘



The text is now cleaned and ready for further processing and analysis!


Text Cleaning Best Practices


While the exact steps and techniques you use for text cleaning will depend on your specific data and goals, here are some general tips and best practices to keep in mind:


    

  • 

    Perform text cleaning steps in a logical order, typically: lowercase → remove numbers → remove punctuation & special characters → tokenize → remove stopwords → stem/lemmatize. Changing the order can impact the results.
    

    • 

  • 

    Consider whether you really need to perform each step. For example, stemming/lemmatization may not be necessary for some tasks. Keeping numbers, punctuation or stopwords may be useful in some cases (e.g. analysing Tweets where # and @ are important).  
    

    • 

  • 

    Always check the output after each cleaning step to avoid unintended data loss or transformation.
    

    • 

  • 

    Stopword removal can be language-specific. Make sure you use the appropriate language corpus.
    

    • 

  • 

    Stemming is faster but lemmatization is more accurate. Depending on your data size and accuracy requirements, choose the appropriate technique.
    

    • 

  • 

    Text cleaning is task-specific. The same text may need to be cleaned differently for different NLP tasks.
    

    • 

  • 

    Expand your text cleaning toolkit beyond NLTK. For example, try stemming/lemmatization with spaCy, Gensim or TextBlob and see if results improve.
    

    • 



Conclusion


We covered a lot of ground in this post! To recap, text cleaning is a crucial preprocessing step for NLP that transforms raw text into a more standardized format. Using NLTK in Python, we can perform various text cleaning tasks like lowercasing, removing irrelevant items like numbers and punctuation, tokenizing, removing stopwords and normalizing word forms through stemming or lemmatization. 


The key is to find the right balance in cleaning your text – clean it enough to remove noise and improve results, but not so much that you lose meaningful information in the process. Hopefully the techniques, code snippets and best practices covered here give you a solid starting point for cleaning your own text data with NLTK. Happy NLP!


      
          
        How useful was this post?      
              
        Click on a star to rate it!      
        
      
                      
  •                       
    •                   
  •                       
    •                   
  •                       
    •                   
  •                       
    •                   
  •                       
    •               
    
    
        
      Average rating 0 / 5. Vote count: 0    
    
      No votes so far! Be the first to rate this post.    
    
  
          



You May Like to Read,

Similar Posts