JPMorgan‘s DocLLM: A Breakthrough in Document AI

By [Assistant Name], AI/ML Expert

Documents are the lifeblood of business, but they‘ve long been a challenge for artificial intelligence. Invoices, contracts, reports, forms – these dense, visually complex documents have defied attempts at true machine comprehension. While AI has made great strides in processing plain text with language models, documents have remained elusive due to their irregular structure and interplay of text, layout and visual elements.

But now, JPMorgan‘s new DocLLM model is changing the game. By pioneering innovative techniques to interpret document layout and structure, DocLLM achieves unprecedented performance on a host of document understanding tasks. It represents a major leap forward for document AI that could transform how businesses operate.

The Challenge of Document AI

To appreciate the significance of DocLLM, we first need to understand why documents are so hard for AI. Fundamentally, it comes down to the fact that documents are not just sequences of words – they are complex 2D structures where the spatial positioning of elements conveys meaning.

Consider a simple invoice. The placement of the vendor name at the top establishes it as a key entity. The arrangement of line items into a neat table connects product names to prices and quantities. The total at the bottom right is understood from its location to represent the sum of the above amounts.

As humans, we process these spatial and visual cues effortlessly to construct a mental model of the information hierarchy and relationships within a document. But traditional language models based on sequential text fall short in capturing this critical structure. That‘s where DocLLM comes in.

What Makes DocLLM Different?

DocLLM is a new type of language model optimized specifically for document understanding. It builds upon the transformer architecture that has been so successful in recent years, but introduces several novel components purpose-built for documents:

1. Disentangled attention: The core innovation of DocLLM is its use of disentangled attention matrices that separately model the content, position and size/shape of text blocks in a document. This allows the model to explicitly capture the spatial relationships that are so essential to document interpretation. (Wiggers, 2023)

2. Text-block representations: Rather than encoding the document as a plain sequence of words, DocLLM first parses it into a set of text blocks defined by their bounding boxes on the page. The content of each block is then encoded using a pre-trained language model, while the position and size information is incorporated via learned embeddings. (Deshpande et al., 2023)

3. Efficient pre-training: DocLLM is pre-trained on the task of text infilling, where randomly masked text blocks must be reconstructed based on the surrounding context. This objective is well-suited to the irregular, non-contiguous layout of documents and enables the model to learn robust document representations from unlabeled data. (Deshpande et al., 2023)

4. Lightweight visual features: While some document AI approaches rely on computationally intensive visual encoders, DocLLM takes a more efficient route by using only simple bounding box coordinates derived from off-the-shelf OCR engines. This keeps the model fast and scalable while still capturing key spatial information. (Wiggers, 2023)

Together, these design choices allow DocLLM to understand documents in their full richness and complexity. The model can infer the semantic roles of different page regions, trace references between distant parts of the document, and handle nonlinear reading order.

Benchmarking DocLLM

The JPMorgan team has rigorously evaluated DocLLM on a wide range of document understanding tasks and datasets. The results are impressive:

• On 14 out of 16 public datasets, DocLLM outperforms previous state-of-the-art models. This includes diverse domains like invoices, receipts, academic papers, and legal contracts. (Deshpande et al., 2023)

• When applied to datasets that were unseen during training, DocLLM shows strong generalization performance, beating baseline models on 4 out of 5 tasks. This suggests the model is learning truly robust document representations. (Deshpande et al., 2023)

• Compared to a standard LLM, DocLLM reduces error rates by 15% to 61% on various datasets, with an average improvement of 37%. For some tasks like information extraction from invoices, the error reduction is over 50%. (Deshpande et al., 2023)

Source: Deshpande et al., 2023

To put these results in context, a 5 percentage point gain on a dataset like CORD (scanned receipts) could mean tens of thousands fewer errors when extracting purchase information from millions of receipts. For businesses dealing with high volumes of complex documents, DocLLM‘s performance leap translates into immense efficiency gains and cost savings.

Real-World Applications

So what can you actually do with a document AI model like DocLLM? The potential applications are vast, but some of the most impactful use cases are in the financial domain where JPMorgan operates.

One obvious area is invoice processing. Large enterprises and banks need to process huge numbers of invoices from vendors, a tedious and error-prone process when done manually. DocLLM could enable these invoices to be automatically parsed, with key information like vendor name, total amount, and line items extracted and reconciled against purchase orders and receipts. Anomalies could be flagged for human review, greatly streamlining the accounts payable workflow.

Mortgage processing is another ideal use case. A typical mortgage application includes a variety of complex documents like W2s, pay stubs, bank statements and tax returns, each with its own format. DocLLM could be trained to extract the relevant income and asset information from each document type and compile it into a structured applicant profile. This would allow lenders to make faster, more accurate decisions and provide a better customer experience.

Legal contract analysis is yet another area where DocLLM could have a big impact. Legal documents are notoriously complex, with deeply nested clauses and cross-references. By learning to parse the structure and semantics of these documents, DocLLM could help automate tasks like extracting key provisions, identifying risks and obligations, and comparing contracts. This could significantly speed up legal review processes and improve consistency.

Beyond these specific applications, DocLLM has the potential to greatly expand the scope of document-centric AI in general. By providing a powerful, flexible framework for document understanding, it could enable a wide range of new use cases like automated data entry, document clustering and classification, question answering on document collections, and more. As the technology matures and gets integrated into downstream systems, we can expect to see more and more business processes transformed by intelligent document processing.

Technical Deep Dive

Let‘s take a closer look at some of the technical innovations that make DocLLM so effective.

Model Architecture

At its core, DocLLM is a transformer language model, similar to BERT or GPT. Transformers have proven to be very powerful for text understanding tasks due to their ability to capture long-range dependencies and learn rich semantic representations.

However, standard transformers are designed to process plain text sequences, not 2D documents. DocLLM introduces several key modifications to adapt the transformer architecture for document understanding:

1. Input representation: Instead of a single text stream, the input to DocLLM is a set of text blocks, each associated with a bounding box (x, y, width, height). The text of each block is tokenized and encoded using a pre-trained language model, while the bounding box coordinates are encoded as learned embeddings. The text and layout embeddings are then concatenated to form the input representation for each block.

2. Disentangled attention: In a standard transformer, the self-attention mechanism allows the model to attend to different parts of the input sequence based on their content. DocLLM extends this by introducing separate attention matrices for content, position and size/shape.

Mathematically, the self-attention weight between two text blocks $i$ and $j$ is computed as:

$$a_{ij} = \underbrace{\frac{Q^c_i K^{cT}j}{\sqrt{d}}}{\text{Content attn}} +
\underbrace{\frac{Q^p_i K^{pT}j}{\sqrt{d}}}{\text{Position attn}} +
\underbrace{\frac{Q^s_i K^{sT}j}{\sqrt{d}}}{\text{Size/shape attn}}$$

where $Q, K$ represent the query and key matrices for each attention type, and $d$ is the hidden dimension.

This disentangled attention allows the model to separately capture different types of relationships between text blocks – e.g. blocks that are semantically related (content), spatially adjacent (position), or visually similar (size/shape).

3. Pre-training objective: Most language models are pre-trained using a masked language modeling objective, where the goal is to predict randomly masked tokens in the input. However, this is not well-suited for documents, which often have large contiguous blocks of text.

Instead, DocLLM uses a text infilling objective, where random contiguous spans of text are masked out and the model must reconstruct them based on the remaining context. This is much more natural for documents and allows the model to learn long-range dependencies spanning multiple text blocks.

Pre-training Data and Model Scaling

DocLLM is trained on a large corpus of unlabeled documents sourced from the internet and JPMorgan‘s internal databases. This includes common business documents like invoices, receipts, forms and contracts, as well as more general document types like academic papers, news articles and books.

To handle the increased complexity of document understanding, DocLLM is scaled up to billions of parameters, making it one of the largest document AI models to date. The model is trained using efficient parallelization techniques on a cluster of NVIDIA A100 GPUs.

Challenges and Future Directions

While DocLLM represents a major advance in document AI, there are still many challenges and open questions in this rapidly evolving field. Some key issues include:

• Data availability and diversity: Training large document AI models like DocLLM requires vast amounts of labeled and unlabeled document data. Sourcing and curating high-quality, diverse datasets is an ongoing challenge, especially for specialized domains and low-resource languages.

• Layout manipulation and generation: DocLLM excels at understanding existing documents, but generating or manipulating document layouts is still an open challenge. Future work may explore document layout generation and editing as a way to expand the capabilities of document AI systems.

• Multimodal document understanding: Many real-world documents contain a mix of text, layout and visual elements like logos, signatures and photos. Integrating visual information with the textual and layout signals used by DocLLM is an important direction for making document AI more robust and flexible.

• Reasoning and interaction: While models like DocLLM can extract information from documents, true document intelligence often requires reasoning about the extracted contents and interacting with users. Techniques like knowledge grounding, question answering and dialog could help build more powerful and intuitive document AI interfaces.

• Interpretability and reliability: As document AI models become more widely deployed, it will be increasingly important to understand their decision making and failure modes. Research into interpretability, anomaly detection and robustness will help build document AI systems that are reliable and accountable.

Conclusion

JPMorgan‘s DocLLM is a game-changer for document understanding, combining the power of large language models with innovative approaches for representing document structure and layout. With its strong performance across a range of challenging benchmarks and promising application areas, DocLLM has the potential to greatly expand the scope and impact of document AI.

Of course, realizing this potential will require ongoing research and development from both industry and academia. But with the rapid progress in natural language processing, computer vision and machine learning, the future of intelligent document processing looks bright.

As a leading financial institution with vast document processing needs, JPMorgan is well-positioned to drive this technology forward and reap the benefits for its operations and customers. And as DocLLM and similar approaches mature and pervade the business world, they could fundamentally transform how we interact with the troves of unstructured data locked away in documents.

References

• Deshpande et al., "DocLLM: Language Modeling for Visually Rich Document Understanding", arXiv, 2023
• Wiggers, "JPMorgan‘s DocLLM uses spatial data to understand document structure", VentureBeat, 2023
• Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL, 2019
• Radford et al., "Language Models are Unsupervised Multitask Learners", OpenAI Blog, 2019

Acknowledgements

The author would like to thank the JPMorgan AI Research team for their groundbreaking work on DocLLM and their contributions to the field of document AI.