Extracting Tabular Data from Microsoft Word Documents Using Python and Machine Learning Techniques

ByJordan Brown

In the world of data science and machine learning, we often take for granted the availability of clean, structured datasets in convenient formats like CSV or JSON. However, in many real-world scenarios, valuable data is locked away in unstructured or semi-structured documents like PDFs, images, web pages, and Microsoft Word files.

Being able to efficiently extract and transform data from these raw formats into a structured representation suitable for analysis is a critical skill in the data professional‘s toolkit. In this article, we‘ll dive deep into the specific challenge of extracting tabular data from Microsoft Word documents using Python and machine learning techniques.

The Challenge of Unstructured Data

Unstructured data, by definition, lacks a predefined data model or organized structure. Word documents are a prime example – they contain a complex mix of formatted text, tables, images and other embedded objects, without any standardized schema or metadata describing the content.

While a human can easily skim a document and identify the relevant pieces of information, automating this process is non-trivial. Simple string parsing techniques quickly fall short when faced with the diversity of layouts, styles and formats found across different documents.

Consider the example of extracting contact information (name, phone, email) from a collection of business documents. A simple approach might search the text for email-like and phone-number-like patterns, but this is prone to false positives and will fail to capture contacts where the key details are separated across different paragraphs or formatting elements.

Machine Learning for Information Extraction

To tackle these challenges, we can leverage machine learning techniques for information extraction and natural language processing. Rather than relying on brittle pattern matching, we can train models to understand the semantics and context of the text, allowing for more robust and flexible extraction.

Some key machine learning approaches for this task include:

  • Named Entity Recognition (NER): NER models are trained to identify and classify named entities like people, organizations, locations, etc. mentioned in the text. We can apply NER to identify contact names.

  • Sequence Labeling: Sequence labeling models assign a categorical label to each token in a text sequence. This allows us to identify entities that span multiple tokens. For example, we could label each token as being part of a name, organization, phone number, email, or other field.

  • Rule-based Parsing: For highly structured text like tables, rule-based parsers can be effective. We define a grammar specifying the expected structure (e.g. rows delimited by newlines, columns by tabs) and use it to parse matching text into a tabular format. Rule-based parsers are less flexible than ML models but can be faster and simpler to implement for narrow domains.

A Python Toolkit for Word Document Processing

To implement these extraction techniques, we‘ll use a combination of open-source Python libraries:

  • python-docx for extracting text and tables from Word documents
  • spaCy for named entity recognition and part-of-speech tagging
  • pandas for structuring extracted data in a tabular format

We‘ll also use some standard Python tools like re for regular expressions and beautifulsoup for handling HTML content.

Here‘s a high-level overview of the extraction pipeline:

  1. Convert Word documents from .doc to .docx format if needed
  2. Extract raw text and tables from the .docx using python-docx
  3. Apply NER model from spaCy to identify contact names
  4. Use regular expressions to extract phone numbers and email addresses
  5. Apply rule-based parser to extract tabular data from identified tables
  6. Merge the extracted entities and tabular data into a pandas DataFrame

Let‘s walk through each of these steps in detail.

Converting Word Documents to .docx Format

The python-docx library only supports .docx format, so we need to convert any legacy .doc files before processing. On Windows, we can automate this using the COM interface:

from win32com import client as wc

def convert_doc_to_docx(doc_path, docx_path):
    word = wc.Dispatch(‘Word.Application‘)
    doc = word.Documents.Open(doc_path)
    doc.SaveAs(docx_path, 16)
    doc.Close()
    word.Quit()

For Linux/MacOS, we can use the libreoffice command-line tool:

import subprocess

def convert_doc_to_docx(doc_path, docx_path):
    cmd = f‘libreoffice --convert-to docx "{doc_path}" --outdir "{os.path.dirname(docx_path)}"‘
    subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Extracting Text and Tables with python-docx

With the document in .docx format, we can use python-docx to extract the raw text and table content:

import docx

def extract_text_and_tables(docx_path):
    doc = docx.Document(docx_path)

    text = []
    for para in doc.paragraphs:
        text.append(para.text)

    tables = []
    for table in doc.tables:
        data = []
        for row in table.rows:
            row_data = []
            for cell in row.cells:
                row_data.append(cell.text)
            data.append(row_data)
        tables.append(data)

    return ‘\n‘.join(text), tables

This gives us the document content as a plain text string and a list of tables, where each table is a list of row lists.

Extracting Entities with Named Entity Recognition

Next, we‘ll apply named entity recognition to identify contact names in the text. We‘ll use the pre-trained English NER model from spaCy:

import spacy

nlp = spacy.load(‘en_core_web_sm‘)

def extract_contact_names(text):
    doc = nlp(text)

    names = []
    for ent in doc.ents:
        if ent.label_ == ‘PERSON‘:
            names.append(ent.text)

    return names

The spaCy model identifies various named entities (people, organizations, locations, etc) which we can access through the doc.ents property. Here we filter for entities labeled as PERSON to extract the contact names.

We can also use spaCy‘s part-of-speech tagging to help identify entities that may not be labeled by the NER model:

def extract_contact_names(text):
    doc = nlp(text)

    names = []
    for token in doc:
        if token.pos_ == ‘PROPN‘ and token.ent_type_ == ‘‘:
            names.append(token.text)

    return names

This code identifies proper nouns (PROPN) that are not part of a named entity, which can help catch contact names missed by the NER model.

Extracting Phones and Emails with Regular Expressions

For identifying phone numbers and email addresses, regular expressions are a simple and effective approach:

import re

def extract_phones(text):
    phone_pattern = re.compile(r‘(\+\d{1,2}\s)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}‘)
    phones = re.findall(phone_pattern, text)
    return [re.sub(r‘\D‘, ‘‘, phone) for phone in phones]

def extract_emails(text):
    email_pattern = re.compile(r‘[\w\.-]+@[\w\.-]+\.\w+‘)
    emails = re.findall(email_pattern, text)
    return emails

These patterns will match common US phone number and email formats. They can be adapted to handle international formats as needed.

Parsing Tables with Regular Expressions

For parsing the extracted Word tables into a structured format, we can use regular expressions to split rows and cells based on expected delimiters. A common format is for rows to be separated by newlines and cells by tabs:

def parse_tables(tables):
    data = []
    for table in tables:
        for row in table:
            row_data = []
            for cell in re.split(r‘t‘, row):
                row_data.append(cell.strip())
            data.append(row_data)
    return data

This assumes that all tables in the document follow the same row/cell delimiter convention. More sophisticated parsing logic may be needed to handle variations across tables.

Putting it All Together

With these extraction functions defined, we can now process a Word document and assemble the extracted data into a structured format:

import pandas as pd

def process_document(doc_path):
    docx_path = doc_path.replace(‘.doc‘, ‘.docx‘)
    convert_doc_to_docx(doc_path, docx_path)

    text, tables = extract_text_and_tables(docx_path)

    contacts = []
    for name in extract_contact_names(text):
        phones = extract_phones(text)
        emails = extract_emails(text)
        contacts.append({‘name‘: name, ‘phone‘: phones[0] if phones else ‘‘, ‘email‘: emails[0] if emails else ‘‘})

    df_contacts = pd.DataFrame(contacts)
    df_tables = pd.DataFrame(parse_tables(tables))

    return df_contacts, df_tables

This function orchestrates the full pipeline:

  1. Converting to .docx
  2. Extracting text and tables
  3. Identifying contact names, phones and emails
  4. Parsing the table data
  5. Structuring the extracted information in pandas DataFrames

We can now easily apply this to a collection of Word documents:

import glob

dfs_contacts = []
dfs_tables = []
for doc_path in glob.glob(‘documents/*.doc‘):
    df_contacts, df_tables = process_document(doc_path)
    dfs_contacts.append(df_contacts)
    dfs_tables.append(df_tables)

df_contacts_all = pd.concat(dfs_contacts)    
df_tables_all = pd.concat(dfs_tables)

This code processes all .doc files in the documents directory and concatenates the extracted DataFrames into combined contact and table DataFrames.

Performance Considerations

Processing a large number of documents can be time and resource intensive, especially if the documents are large or complex. Some strategies for optimizing performance include:

  • Use multiprocessing to parallelize document processing across multiple CPU cores
  • Extracting text and tables is the most expensive step, so do this processing once and cache the results for subsequent runs
  • Avoid loading the entire document into memory at once. Instead, use python-docx‘s incremental parsing functionality to process the document in chunks.
  • Distribute processing across multiple machines using a tool like Apache Spark or Dask for very large document collections.

Integrating with a Machine Learning Pipeline

The document processing functions we‘ve developed can be integrated into a larger machine learning pipeline for tasks like document classification, information extraction, or search and recommendation.

For example, we could use the extracted contact information as features for a model that predicts the business category of a given document. Or we could use the extracted tables to build a searchable database of product information.

Some key considerations when integrating document processing into an ML pipeline:

  • Ensure consistent data formatting and handle missing or malformed data
  • Use feature scaling and normalization to account for differences in document length and content
  • Split document collections into training, validation and test sets for model evaluation
  • Monitor pipeline performance over time and retrain models on new data as needed

Advanced python-docx Functionality

In addition to basic text and table extraction, python-docx provides a rich set of functionality for creating and manipulating Word documents, including:

  • Paragraphs: adding, styling and formatting text content
  • Sections: setting page size, orientation, margins, etc.
  • Tables: inserting and populating complex table structures
  • Images: embedding images in the document
  • Styles: managing paragraph, character, list and table styles
  • Headers and Footers: adding and editing headers and footers
  • Hyperlinks: inserting and managing hyperlinks
  • Bookmarks: creating and referencing bookmarks within the document

Here‘s an example of using python-docx to create a document with a table of extracted contact information:

from docx import Document
from docx.shared import Inches

def create_contact_report(df_contacts, output_path):
    doc = Document()

    doc.add_heading(‘Contact Report‘, 0)

    table = doc.add_table(rows=df_contacts.shape[0]+1, cols=df_contacts.shape[1])

    header_row = table.rows[0]
    for i, col_name in enumerate(df_contacts.columns):
        header_row.cells[i].text = col_name

    for i, row in df_contacts.iterrows():
        for j, value in enumerate(row):
            table.rows[i+1].cells[j].text = str(value)

    doc.save(output_path)

This creates a new document, adds a main heading, and then inserts a table populated with the values from the contact DataFrame. Additional formatting could be applied as needed.

Other Python Libraries for Word Documents

While python-docx is a popular and full-featured library, there are other Python packages that provide Word document processing capabilities:

  • docx2txt: a simple package focused on text extraction
  • olefile (formerly OleFileIO_PL): a low-level package for reading and writing Microsoft OLE2 files (the legacy .doc format)
  • textract: a package that provides a unified interface for extracting text from various file formats, including .doc and .docx
  • mammoth: a package for converting .docx documents to HTML or Markdown

The choice of library will depend on your specific requirements around document types, processing complexity, and output formats.

Conclusion and Further Reading

Extracting structured data from Word documents is a common and challenging task in many data workflows. By leveraging Python tools like python-docx and spaCy, and applying techniques from natural language processing and machine learning, we can develop robust and scalable document processing pipelines to transform unstructured data into valuable insights.

Some additional resources for further learning:

As the volume and variety of unstructured business data continues to grow, skills in document processing and information extraction will only become more valuable. I encourage you to experiment with the techniques covered in this article and find opportunities to apply them in your own projects!

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts