The Ultimate Regex Cheatsheet for Natural Language Processing Tasks

Regular expressions, or regex for short, are an essential tool in any natural language processing (NLP) practitioner‘s toolkit. Regex allows you to define search patterns to find, extract, replace, and manipulate text. With the power and flexibility of regex, you can tackle a wide variety of text processing tasks with just a few lines of code.

In this ultimate guide, we‘ll share some of the most useful regex patterns for common NLP tasks. Whether you need to extract emails and phone numbers, remove HTML tags, or tokenize text, these tried-and-true recipes will help you get the job done quickly. We‘ll walk through realistic examples in Python using the built-in re module.

Let‘s start by reviewing some of the key functions in Python‘s re library that you‘ll want to get familiar with:

• re.findall – Finds all matches of a pattern in a string and returns them as a list
• re.sub – Substitutes all occurrences of a pattern in a string with a replacement
• re.match – Checks if a pattern matches the beginning of a string
• re.search – Searches a string for the first location where a pattern matches

We‘ll primarily be using re.findall in our examples to find and extract matches. Now let‘s dive into some handy regex patterns organized by category.

Text Cleaning

A big part of NLP involves cleaning and normalizing text data. Here are a few regex tricks for common text cleaning operations:

Remove non-alphanumeric characters

def remove_special_chars(text):
    return re.sub(r‘[^a-zA-Z0-9]‘, ‘ ‘, text)

text = "Hello! This string has #hashtags, @mentions, and $ymbol$."
print(remove_special_chars(text))

Output:

Hello  This string has  hashtags   mentions  and  ymbol  

The regex pattern [^a-zA-Z0-9] will match any character that is not a letter or number. We use re.sub to substitute these matches with a space character.

Remove extra whitespace

def remove_extra_whitespace(text):
    return re.sub(r‘\s+‘, ‘ ‘, text)

text = "This     string has   a      lot of extra    whitespace."
print(remove_extra_whitespace(text))  

Output:

This string has a lot of extra whitespace.

The regex \s+ will match one or more whitespace characters (space, tab, newline). Using re.sub, we can replace these matches with a single space to normalize the text.

Remove HTML tags

def remove_html_tags(text):
    return re.sub(‘<.*?>‘, ‘‘, text)

text = "This text contains <b>HTML</b> <a href=‘https://example.com‘>tags</a>."
print(remove_html_tags(text))

Output:

This text contains HTML tags.

The pattern <.*?> matches HTML tags by looking for text surrounded by angle brackets. The ? makes it a "lazy" match that stops at the first closing bracket.

Text Extraction

Extracting specific pieces of information from unstructured text is a common NLP application. Here are some useful regex patterns for finding and pulling out key bits of text:

Extract URLs

def find_urls(text):
    url_pattern = r‘https?://\S+‘
    urls = re.findall(url_pattern, text)
    return urls

text = "Check out my blog at https://www.example.com for more articles like this! You can also find my code repos at https://github.com/johndoe."
print(find_urls(text))

Output:

[‘https://www.example.com‘, ‘https://github.com/johndoe‘]

This regex will find URLs starting with "http://" or "https://" followed by any non-whitespace characters.

Extract email addresses

def find_email_addresses(text):
    email_pattern = r‘\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b‘
    emails = re.findall(email_pattern, text)
    return emails

text = "Contact us at [email protected] or [email protected] for assistance."
print(find_email_addresses(text))  

Output:

[‘[email protected]‘, ‘[email protected]‘]

This comprehensive pattern will match most valid email addresses, allowing for a range of special characters in the username portion.

Extract phone numbers

def find_phone_numbers(text):
    phone_pattern = r‘\b\d{3}[-.]?\d{3}[-.]?\d{4}\b‘
    phone_numbers = re.findall(phone_pattern, text) 
    return phone_numbers

text = "Call 212-555-0191 or 415.123.4567 for more information."
print(find_phone_numbers(text))

Output:

[‘212-555-0191‘, ‘415.123.4567‘] 

This pattern will find 10-digit US phone numbers with optional dashes or periods between the number groups. You can modify it for phone numbers in other formats.

Extract prices

def find_prices(text):
    price_pattern = r‘\$\d+(\.\d{2})?‘
    prices = re.findall(price_pattern, text)
    return prices

text = "The shirt costs $19.99 and the pants cost $34."
print(find_prices(text))

Output:

[‘$19.99‘, ‘$34‘]

The regex looks for a dollar sign followed by one or more digits, optionally followed by a decimal point and two digits. It will find prices formatted as "$5" or "$5.99".

Text Manipulation

Once you‘ve extracted the text you need, the next step is often to transform or manipulate it in some way. Regex can be a useful tool for tasks like replacing substrings or splitting text into tokens.

Remove punctuation

def remove_punctuation(text):
    return re.sub(r‘[^\w\s]‘, ‘‘, text) 

text = "I can‘t believe this! Did she really say that? OMG!!!"
print(remove_punctuation(text))

Output:

I cant believe this Did she really say that OMG

The character class [^\w\s] will match anything that‘s not a word character or whitespace. We use it with re.sub to remove all punctuation from the input text.

Split text into sentences

def split_into_sentences(text):
    sentence_pattern = r‘[^.!?]+[.!?]‘
    sentences = re.findall(sentence_pattern, text)
    return sentences

text = "Hello there! How are you doing today? I hope you‘re having a great day."
print(split_into_sentences(text))

Output:

[‘Hello there!‘, ‘How are you doing today?‘, "I hope you‘re having a great day."]

This pattern matches sentences by finding stretches of text that don‘t contain a period, exclamation mark, or question mark, followed by a punctuation character. The result is a list of sentences extracted from the paragraph.

Tokenize text into words

def tokenize_text(text):
    word_pattern = r‘\w+‘
    words = re.findall(word_pattern, text)
    return words

text = "The quick brown fox jumped over the lazy dog."  
print(tokenize_text(text))

Output:

[‘The‘, ‘quick‘, ‘brown‘, ‘fox‘, ‘jumped‘, ‘over‘, ‘the‘, ‘lazy‘, ‘dog‘]

The regex \w+ matches sequences of word characters (letters, numbers, underscores). Using it with re.findall allows us to split a text into individual word tokens.

Advanced NLP

Regex can also be used for more niche NLP tasks that require identifying specific types of entities or expressions in text. Here are a few examples:

Extract acronyms and initialisms

def find_acronyms(text):
    acronym_pattern = r‘\b[A-Z]{2,}\b‘
    acronyms = re.findall(acronym_pattern, text)
    return acronyms

text = "I work for NASA, not the FBI or CIA. BTW, do you know what POTUS stands for?"
print(find_acronyms(text))  

Output:

[‘NASA‘, ‘FBI‘, ‘CIA‘, ‘BTW‘, ‘POTUS‘]

This pattern simply looks for words in all capital letters that are at least two characters long. It‘s an easy way to pull out many acronyms and initialisms.

Match proper names

def find_proper_names(text):
    name_pattern = r‘\b([A-Z]\w+\s+(?:[A-Z]\w+\s+)?[A-Z]\w+)\b‘
    names = re.findall(name_pattern, text)
    return names

text = "Notable figures like Barack Obama, George W. Bush, and Neil deGrasse Tyson have all contributed to science in different ways."
print(find_proper_names(text))

Output:

[‘Barack Obama‘, ‘George W. Bush‘, ‘Neil deGrasse Tyson‘]

This slightly more complex regex will match most two and three word proper names. It looks for a capital letter followed by one or more word characters, then one or two additional words with the same pattern.

Identify hashtags and mentions

def find_hashtags_and_mentions(text):
    hashtag_pattern = r‘#\w+‘
    mention_pattern = r‘@\w+‘
    hashtags = re.findall(hashtag_pattern, text)
    mentions = re.findall(mention_pattern, text)
    return hashtags, mentions

text = "I‘m so excited for the #MachineLearning conference! I‘ll be speaking with @JaneDoe about #NLP."
hashtags, mentions = find_hashtags_and_mentions(text)
print("Hashtags:", hashtags)
print("Mentions:", mentions)

Output:

Hashtags: [‘#MachineLearning‘, ‘#NLP‘]
Mentions: [‘@JaneDoe‘]

Hashtags are identified by the pattern #\w+ – a pound sign followed by any word characters. Similarly, mentions are an @ sign followed by word characters. These patterns make it easy to extract these entities commonly found in social media posts.

Conclusion

As you can see, regular expressions provide a concise and powerful way to find and manipulate text for a variety of NLP tasks. While they can seem cryptic at first, building up a toolbox of go-to regex patterns will let you clean, extract, and transform text data with ease.

The examples covered here are just the tip of the iceberg – there are an endless number of ways to combine and modify regular expressions to suit your particular text processing needs. Regex is an invaluable tool to master for anyone working with natural language data.

I hope this cheatsheet has given you some practical regex recipes to add to your NLP toolkit. Try them out in your own projects and don‘t be afraid to experiment and create your own patterns. With practice and experience, you‘ll be able to quickly craft custom regex solutions for all sorts of linguistic challenges.

Happy text processing!