Detecting Table Rows and Columns in Images Using Transformers

ByJordan Brown

Tables are an incredibly efficient and common way to present data in documents. But while tables are easily interpretable by humans, extracting data from tables in unstructured documents like PDFs or images is a major challenge for machines. The huge variety of table layouts, formats, and styles make it difficult to programmatically identify and parse tabular data at scale.

Fortunately, recent advances in deep learning are providing new solutions to these old problems. Transformers, a neural network architecture first developed for natural language processing (NLP), are now being applied to computer vision tasks with impressive results. In this post, we‘ll take a deep dive into how transformers work and how they‘re being used to detect and recognize the structure of tables in documents.

The Challenge of Table Extraction

Before we get into the technical details of transformers, let‘s first understand why extracting tables from documents is so difficult. There are a few key challenges:

  1. Layout variety: Tables can have wildly different layouts and formats. Think of all the tables you‘ve seen – simple grids, multi-level headers, merged cells, nested tables within tables. This variety makes it impossible to use simple heuristics or rules to reliably identify table structures.

  2. Lack of standard annotations: Unlike natural images which have large, standardized datasets for object detection and segmentation (e.g. COCO, ImageNet), there is no universal dataset for table extraction. This lack of labeled data makes it challenging to train robust machine learning models.

  3. Ambiguity of cell contents: Table cells can contain a mix of text, numbers, and even graphics or sub-tables. This makes it difficult to distinguish headers from data and infer the semantic meaning of each cell.

Despite these challenges, tables contain a wealth of valuable information across many domains. A 2021 study found that scientific papers contain an average of 4.4 tables per document. In the financial domain, over 200 million PDFs are published each year, many containing tables with essential data. Being able to automatically extract and structure this tabular data would enable analysis and insights at an unprecedented scale.

Transformers: From Language to Vision

Transformers were first introduced in the 2017 paper "Attention Is All You Need" as a new architecture for machine translation. The key innovation of transformers was the self-attention mechanism, which allowed the model to attend to different parts of the input sequence and learn long-range dependencies.

Transformers quickly took over the field of NLP, achieving state-of-the-art results on tasks like language understanding, text generation, and question answering. But it didn‘t take long for researchers to recognize their potential for computer vision tasks as well.

In 2020, Facebook AI Research (FAIR) released the Detection Transformer (DETR), which adapted the transformer architecture for object detection in images. DETR framed object detection as a direct set prediction problem – given an input image, the model predicts a fixed-size set of bounding boxes and class labels for each object in the image.

DETR architecture
The DETR architecture. Image source: Papers With Code

Under the hood, DETR uses a convolutional neural network (CNN) backbone to extract visual features from the input image. These features are then flattened and passed into a transformer encoder-decoder architecture. The transformer encoder learns a compact representation of the image that captures the global context, while the decoder attends to this representation to predict the object bounding boxes and classes.

One of the key advantages of DETR is its simplicity and flexibility compared to previous object detection models like Faster R-CNN. DETR eliminates the need for complex components like region proposal networks and non-maximum suppression. It also enables end-to-end training with a bipartite matching loss function that uniquely assigns predicted boxes to ground truth boxes.

DETR achieved state-of-the-art results on the popular COCO object detection benchmark. But more importantly, it provided an elegant framework for detecting more complex structures like tables, as we‘ll see next.

PubTables-1M: A Million Tables for Training

To train a transformer model to accurately detect tables and their structure, you need a lot of labeled training data. That‘s where the PubTables-1M dataset comes in.

Published by Microsoft Research in 2021, PubTables-1M is a dataset of over 1 million tables extracted from scientific articles across domains like medicine, biology, and computer science. It‘s by far the largest public dataset for table extraction and structure recognition, with rich annotations including:

  • Table bounding boxes
  • Table row and column locations
  • Cell bounding boxes and their row/column indexes
  • Cell content type (header, metadata, data)
  • Footnotes and references

Here are some key statistics on PubTables-1M:

  • 948,000+ tables from 270,000+ documents
  • Covers tables in PDF, LaTex, and HTML formats
  • 5.6M individual table cells annotated
  • 91% of tables have <=10 rows, 93% have <=10 columns
  • 40.8% of cells are headers, 57.9% are data, 1.2% metadata

One unique aspect of PubTables-1M is that the raw annotations went through a cleaning process called canonicalization. The authors found many errors in the original annotations, specifically over-segmentation of table cells. The canonicalization algorithm fixed these by merging cells to align with the true logical structure of the table.

PubTables canonicalization
Example of table canonicalization in PubTables-1M. Image source: Hello Paperspace Blog

Experiments showed that training on the canonicalized data improved table extraction accuracy by 4-6% over the raw annotations, confirming the importance of high-quality training data. PubTables-1M has quickly become the standard benchmark for evaluating table extraction models.

Table Transformers for Structure Recognition

Building on the success of DETR for object detection and the availability of PubTables-1M, researchers at Microsoft developed the Table Transformer – a model specifically designed for recognizing the row and column structure of tables from images.

The architecture of the Table Transformer closely follows that of DETR. The input table image is first passed through a CNN backbone (in this case a ResNet-101) to generate a 2D feature map. This feature map is then flattened and combined with learned positional encodings before being fed into the transformer encoder.

The transformer encoder uses self-attention to capture the global context and spatial relationships within the image. The output of the encoder is a compact 2D representation that preserves the spatial structure of the input.

This representation is then passed to the transformer decoder, which uses cross-attention to map the encoded features to a set of output predictions. For table structure recognition, these predictions correspond to the bounding boxes of each table cell, arranged into rows and columns.

During training, the predicted cell boxes are matched to ground truth annotations using the same bipartite matching loss as DETR. This enables the model to be trained end-to-end to directly output the structured table.

Table Transformer predictions
Example Table Transformer predictions. Image source: Hello Paperspace Blog

The Table Transformer achieves impressive results on the PubTables-1M benchmark. For table structure recognition, it obtains a 94.3% F1 score, significantly outperforming earlier rules-based and heuristic methods. It also maintains high accuracy on more challenging tasks like functional analysis (classifying cell roles), with an 86% F1 score.

Detailed results are shown in the table below. The metrics used are precision (P), recall (R), and F1 score:

Method Table Detection Table Structure Recognition Functional Analysis
Table Transformer 0.993 / 0.995 / 0.994 0.954 / 0.932 / 0.943 0.872 / 0.848 / 0.860
Heuristic (connected components) 0.969 / 0.953 / 0.961 0.816 / 0.781 / 0.798 –

The strong results of the Table Transformer demonstrate the power of the end-to-end approach enabled by transformers. By learning to directly output the structured table from the image, the model avoids the compounding errors of multi-stage pipelines and the need for manual heuristics.

Implementing Table Transformers

While the research behind Table Transformers is complex, using them is quite straightforward thanks to open-source implementations. The Table Transformer is available in the Hugging Face model hub, with pre-trained weights for both table detection and structure recognition.

Here‘s a minimal example of performing inference with the structure recognition model in Python:

from transformers import DetrFeatureExtractor, TableTransformerForObjectDetection
from PIL import Image
import torch

feature_extractor = DetrFeatureExtractor.from_pretrained("microsoft/table-transformer-structure-recognition")
model = TableTransformerForObjectDetection.from_pretrained("microsoft/table-transformer-structure-recognition")

image = Image.open("table.png")
inputs = feature_extractor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs) 

predicted_boxes = outputs.pred_boxes.squeeze()

# Display predicted bounding boxes on image
draw = ImageDraw.Draw(image) 
for box in predicted_boxes:
  box = box.tolist()
  draw.rectangle(box, outline ="red")

image.show()

This code loads the pre-trained Table Transformer model and feature extractor, then runs inference on an input table image. The predicted cell bounding boxes are extracted from the model output and visualized by drawing them on the original image.

Of course, this is just a starting point. A full table extraction pipeline would also need to handle detecting tables on a larger page or document, recognizing the text within each cell using OCR, and exporting the extracted table data to a structured format like CSV or JSON.

The Future of Table Extraction

Table Transformers represent an exciting step forward for table extraction and structure recognition. But there are still many challenges to solve before we have truly robust and generalizable table extraction systems.

One key area for improvement is handling more complex table layouts. While PubTables-1M covers a variety of table structures, it‘s still limited to tables from scientific publications. Tables in other domains like finance or law may have even more irregular layouts that break the assumptions of current models.

Another important direction is integrating table structure recognition with optical character recognition (OCR) to extract not just the locations of table cells but also their textual content. This requires careful coordination between the visual table parsing and language understanding components.

There‘s also more work to be done in evaluation and benchmarking. While PubTables-1M provides a strong starting point, the table extraction community still lacks the kind of standardized, diverse benchmarks that have driven fields like object detection. Developing these benchmarks will be key to measuring progress.

From an application perspective, the potential impact of table extraction is enormous. Imagine a financial analyst being able to automatically extract and analyze data from thousands of quarterly reports, or a medical researcher cross-referencing results across clinical trial publications. Reliable table extraction would enable data aggregation and analysis at an unprecedented scale.

As transformers and other neural network architectures continue to advance, we can expect to see rapid progress in table extraction in the coming years. Combined with the growing availability of large-scale labeled datasets like PubTables-1M, there‘s a bright future ahead for automating the tedious but critical task of extracting data from unstructured documents.

Conclusion

Let‘s recap some of the key points we‘ve covered:

  • Tables are a ubiquitous and valuable source of data across industries, but extracting that data remains a challenge due to the diversity of table layouts and formats.

  • Transformers, originally developed for natural language processing, are proving to be a powerful tool for computer vision tasks like object detection and now table structure recognition.

  • The PubTables-1M dataset, with over 1 million annotated tables from scientific articles, has become a key benchmark for developing and evaluating table extraction models.

  • Table Transformers build on the Detection Transformer (DETR) architecture to directly predict the row and column structure of tables in images, achieving state-of-the-art results on PubTables-1M.

  • While Table Transformers are an important step forward, there are still many challenges to solve in table extraction, including handling more diverse table layouts, integrating with OCR, and developing standardized benchmarks.

The field of document AI is advancing rapidly, and table extraction is one of the key frontiers. With continued research and development of transformer-based models and large-scale datasets, we‘re moving closer to a future where extracting structured data from unstructured documents is a solved problem.

References and Further Reading

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts