[‘The‘, ‘quick‘, ‘brown‘, ‘fox‘, ‘can‘, ‘t‘, ‘jump‘, ‘32‘, ‘3‘, ‘feet‘, ‘right‘]

Regular expressions are a concise and flexible tool for matching patterns in strings, with extensive applications in natural language processing. Whether you need to clean messy text data, extract structured information from documents, or translate between data formats, regular expressions should be a foundational part of your NLP skill set.

In this comprehensive guide, we‘ll build your regex skills from the ground up, beginning with the basic components of regular expressions and progressing to sophisticated techniques for matching complex patterns in text. Throughout the guide, we‘ll use Python‘s re module to show you how to apply regular expressions to practical NLP problems.

Fundamentals of Regular Expressions

At its core, a regular expression is just a sequence of characters that defines a pattern. When you search a string with a regex, you‘re asking: "Does this string contain a substring that matches the pattern?"

The simplest type of pattern is a literal — the exact sequence of characters you want to match. To match the literal string "the", you can use this regex:

the

A regex pattern can also contain metacharacters that give you more flexibility to match different types of strings. Here are a few of the most important types of metacharacters:

. (dot): Matches any single character except a newline

• (star): Matches the preceding character or subexpression zero or more times

• (plus): Matches the preceding character or subexpression one or more times
  ? (question mark): Makes the preceding character or subexpression optional
  ^ (caret): Matches the start of a string
  $ (dollar sign): Matches the end of a string

Character classes allow you to specify a set of characters to match. For example, [aeiou] will match any lowercase vowel. You can also specify a range of characters, like [a-z] to match any lowercase ASCII letter or [0-9] to match any digit.

Combining literals, metacharacters, and character classes allows you to construct versatile patterns. This regex will match a string starting with a capital letter, followed by three lowercase letters, then an optional space and digit:

^[A-Z][a-z]{3}[ ]?[0-9]$

The {3} is a quantifier that matches exactly three occurrences of the preceding subexpression (in this case [a-z]). The ? makes the space character optional, and the ^ and $ anchor the pattern to the start and end of the string.

Regex for Common NLP Tasks

Let‘s see how to use regular expressions for some routine natural language processing tasks in Python.

Tokenization involves splitting text into individual words or tokens. Here‘s a regex that matches sequences of word characters (letters, digits, and underscores), splitting on any non-word characters:

import re

text = "The quick brown fox can‘t jump 32.3 feet, right?"
tokens = re.findall(r"\w+", text)

print(tokens)

Text data frequently contains extra whitespace, punctuation, or other characters that need to be removed before further processing. This code snippet lowercases the text, then strips any non-alphanumeric characters from the start or end of each token:

clean_tokens = [] for token in tokens:
clean = re.sub(r"^\W+|\W+$", "", token.lower())
clean_tokens.append(clean)

print(clean_tokens)

To extract emails, phone numbers, URLs, and other valuable structured data from plain text, regular expressions are indispensable. This example finds email addresses by matching strings containing a username, @, domain name, and top-level domain:

text = "Contact me at [email protected] or [email protected]"

emails = re.findall(r"[a-z0-9._-]+@[a-z0-9.-]+.[a-z]{2,}", text, flags=re.I)

print(emails)

The flags=re.I argument makes the regex case-insensitive. The {2,} quantifier matches two or more lowercase letters for the top-level domain.

Advanced Regex Techniques

Capturing groups allow you to retrieve the parts of a string that matched a particular portion of your regex. Wrapping a subexpression in parentheses creates a capturing group. You can backreference these captures in your pattern or access them in the match object.

This example parses "lastname, firstname" names into their components:

name = "Doe, Jane"
match = re.match(r"^(\w+), (\w+)$", name)
last, first = match.groups()

print(f"{first} {last}")

Lookahead and lookbehind assertions allow you to match a pattern only if it‘s followed or preceded by another pattern, without including that other pattern in the match. Positive lookaheads are denoted by (?=…) and negative lookaheads by (?!…). Positive lookbehinds use (?<=…) and negative lookbehinds use (?<!…).

Let‘s find all words before a comma, but exclude the comma from our match:

text = "Please buy: apples, bananas, bread, milk"
print(re.findall(r"\w+(?=,)", text))

The regex \w+(?=,) matches one or more word characters only if they‘re immediately followed by a comma, and doesn‘t consume the comma.

Using Regex Efficiently in Python

To avoid recompiling the same regex pattern over and over, you can compile it once with re.compile() and then call its methods:

emailregex = re.compile(r"[a-z0-9.-]+@[a-z0-9.-]+.[a-z]{2,}", flags=re.I)

for text in many_strings:
matches = email_regex.findall(text)
…

Sometimes your regexes may contain a lot of special characters, making them difficult to read. You can add comments, whitespace, and line breaks to a verbose regex by passing re.VERBOSE or re.X as a flag:

isbn_regex = re.compile(r"""
^
(?:ISBN(?:-1[03])?:?[ ])? # optional ISBN/ISBN-13 prefix
(?=[-0-9 ]{17}$) # must be 17 chars including hyphens
97[89]-? # ISBN-13 prefix
[0-9]{1,5}-? # group identifier
[0-9]+[-]?[0-9]+[-]? # publisher and title identifiers
[0-9] # check digit
$
""", re.VERBOSE)

This monster of a regex can validate ISBN-13 numbers, allowing for optional hyphens and spaces. The (?:…) non-capturing group matches an optional ISBN/ISBN-13 prefix at the start. The (?=…) lookahead asserts that the string contains exactly 17 characters including hyphens. The rest of the pattern matches the various components of a valid ISBN.

Exercises and Further Resources

Regular expressions take practice to master. Here are some exercises you can work through to hone your regex skills:

1. Write a regex to validate a US phone number in the format (123) 456-7890.

2. Extract all Twitter mentions (strings starting with @) and hashtags (strings starting with #) from a tweet.

3. Find all dates in YYYY-MM-DD format in a string.

4. Write a regex to validate an IP address.

5. Match strings containing only balanced pairs of parentheses, like "(())()" but not "(()" or ")(".

For more practice, try solving the regex exercises on HackerRank or RegexOne. For a deeper dive into regular expressions in Python, check out the Python Regular Expression HOWTO in the official docs. Mastering Regular Expressions by Jeffrey Friedl is a comprehensive book on regexes. And RegexBuddy is a helpful tool for interactively building and testing regular expressions.

Conclusion

We‘ve covered a lot of ground in this guide, from the basics of regular expression syntax to advanced techniques for matching complex patterns in Python. You should now have a solid foundation for using regular expressions to tackle a variety of natural language processing tasks.

Remember, the best way to get comfortable with regular expressions is through practice. Challenge yourself to solve regex problems, and experiment with regexes in your own NLP projects. With time and experience, you‘ll be able to quickly craft concise and powerful regexes to match any pattern you need.