# Extracting Information from Reports Using Regular Expressions in Python - Canonical: https://33rdsquare.com/extracting-information-from-reports-using-regular-expressons-library-in-python/ - Published: 2024-09-03 - Author: Jordan Brown - Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/) --- ## Introduction As a data scientist or analyst, you often need to extract key pieces of information from unstructured text data such as reports, articles, logs, and customer feedback. This could involve pulling out things like names, dates, ID numbers, dollar amounts, email addresses, and so on. Manual scanning and copying is tedious and error-prone, especially for large volumes of text. Fortunately, regular expressions provide a powerful and flexible way to automate the extraction of text that matches a specific pattern. Regular expressions, or regex for short, define a search pattern for strings. You can then search for matches to the pattern within a larger body of text. Python offers excellent support for regular expressions through its built-in re module. In this article, we‘ll dive into how you can leverage regex in Python to extract information from text reports. We‘ll cover the essential concepts of regular expressions, walk through examples of common patterns, and explore tools to help you develop and debug your own regex patterns with ease. While there are other approaches to information extraction, such as looking up keywords in a pre-defined dictionary or applying machine learning models, regex is a powerful and commonly used technique, especially when the information follows a consistent pattern. It‘s an essential tool in the text mining toolbox. ## The re Module in Python To get started with regular expressions in Python, you first need to import the re module: import re When defining regex patterns in Python code, it‘s recommended to use raw string notation by prefixing the string with r. This treats backslash as a literal character, which is very common in regex patterns. pattern = r‘\d+‘ For improved performance, you can compile the regex pattern into a regex object using re.compile(). You can then call methods like search() and findall() on the compiled object. regex = re.compile(r‘\d+‘) matches = regex.findall(text) Here are some of the key functions in the re module: - re.search(pattern, string) – finds the first location where the regex pattern matches the string - re.findall(pattern, string) – finds all matches and returns them as a list of strings - re.sub(pattern, repl, string) – substitutes matches with the specified replacement string The re module also provides flags to control the behavior of the regex engine, such as re.IGNORECASE to perform a case-insensitive match and re.VERBOSE to allow comments and whitespace in the pattern for improved readability. ## Regex Syntax and Pattern Matching Regex patterns are composed of normal characters as well as special metacharacters and sequences. Here are some of the key elements of regex syntax: - . (dot) – matches any character except a newline - \d – matches any digit, equivalent to [0-9] - \w – matches any word character (alphanumeric & underscore), equivalent to [a-zA-Z0-9_] - \s – matches any whitespace character (space, tab, newline) - - - matches zero or more of the preceding character/group - - - matches one or more of the preceding character/group - ? – matches zero or one of the preceding character/group - {n} – matches exactly n of the preceding character/group - {n,m} – matches between n and m of the preceding character/group - ^ – matches the start of a string or line - $ – matches the end of a string or line - [] – defines a character class, e.g. [aeiou] matches any vowel - | – matches either the expression before or after, like an OR operator - () – defines a capturing group and allows applying quantifiers to the entire group Let‘s look at a few examples: - \d{3}-\d{2}-\d{4} matches a Social Security Number like 123-45-6789 - [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-z]{2,} matches an email address - ((\d{3})\s*)?\d{3}[-.]?\d{4} matches US phone numbers in various formats Quantifiers like _and + are greedy by default, meaning they will match as much as possible. You can make them non-greedy by following with a ?, as in_? or +?, to match as little as possible. ## Common Use Cases and Examples Now let‘s walk through some practical examples of using regex to extract different types of information from text. ### Extracting Email Addresses [\w.-]+@[\w.-]+.[\w]{2,} This pattern matches one or more word characters, dots or hyphens, followed by @, followed by a domain name of word characters, dots or hyphens, followed by a dot and then two or more word characters for the top-level domain. It would match email addresses like: - john.smith@company.com - jane_doe+shopping@mail.blog - me@example.co.uk ### Extracting Phone Numbers (+\d{1,2}\s)?(?\d{3})?[\s.-]\d{3}[\s.-]\d{4} This pattern optionally matches an international calling code like +1, then an optional area code with parens, then 3 digits, separator, 3 digits, separator, 4 digits. It would match phone numbers like: - +1 (212) 555-3456 - 415.123.4567 - 212-555-0101 ### Extracting Dates \d{1,2}[/-]\d{1,2}[/-]\d{2,4} This pattern matches a date in MM/DD/YYYY or DD-MM-YY type format. It allows 1-2 digits for day and month and 2-4 digits for year. It would match dates like: - 04/29/2023 - 25-12-22 Note that without additional validation, this pattern may match some invalid dates like 99/99/9999. More complex expressions can be written to handle different separators and formats. ### Extracting Account Numbers \d{4}([- ]\d{4}){3} This pattern matches 4 digits, followed by a group of separator and 4 digits, repeated 3 times. It would match 16-digit account numbers like: - 1234 5678 9012 3456 - 9876-5432-1098-7654 ### Adding Linked References Let‘s say you have a report with citations or references to other documents in a standard format, like: According to the company‘s latest annual report [ACME 2022], profits have increased by 25% year-over-year. We can use a regex substitution to wrap the citation in a hyperlink: re.sub(r‘[([A-Z]+ \d{4})]‘, r‘[\1](https://33rdsquare.com/path/to/1.pdf)‘, text) The \1 in the replacement string refers to the captured group in the search pattern. This would turn [ACME 2022] into [ACME 2022](https://33rdsquare.com/path/to/ACME%202022.pdf). ## Tools for Regex Development Regex patterns can get quite complex, making them challenging to develop and debug. Fortunately, there are several tools that can help. Regex testers like [regex101](https://regex101.com/) and [RegExr](https://regexr.com/) provide an interactive environment to develop and test your patterns. You can enter a regex pattern and a sample text, and it will highlight the matches. They also provide explanations for the different parts of the pattern and a quick reference guide. To understand how the different parts of a regex pattern work together and what they‘ll match, tools like [Regexper](https://regexper.com/) generate railroad diagrams to visualize the logic. This is especially helpful for deciphering long and complex patterns written by someone else. Most code editors and IDEs like VS Code, PyCharm, Sublime Text, etc. have built-in support for searching and replacing text using regular expressions. They also provide syntax highlighting for regex patterns. ## Conclusion As we‘ve seen, regular expressions are an incredibly powerful and versatile tool for working with text data in Python. With a basic understanding of regex syntax and the re module, you can extract all sorts of structured data from unstructured reports and documents. While regular expressions have a reputation for being cryptic and difficult to learn, they are an essential skill for any data professional who deals with text data. Investing time to learn regexes will pay dividends throughout your career, allowing you to accomplish complex text manipulation and extraction tasks with just a few lines of code. As a fun challenge, see if you can come up with a regex pattern to match CrossWord puzzle clues and answers, like: 1 Across – "Monty Python" arachnid (6 letters) 3 Down – Fermented honey drink (4 letters) ANSWERS: PYTHON MEAD Happy regex learning! --- Source: [Extracting Information from Reports Using Regular Expressions in Python](https://33rdsquare.com/extracting-information-from-reports-using-regular-expressons-library-in-python/)