A Comprehensive Guide to Invoice Label Detection using OpenCV

Introduction

Invoice label detection is an important application of computer vision and optical character recognition (OCR) that allows for automatically extracting key information from invoice images, such as the invoice number, date, vendor name, total amount, and more. By automating this process of identifying and extracting structured data from unstructured invoice documents, businesses can greatly reduce manual data entry, save time, and improve efficiency.

In this in-depth guide, we‘ll walk through how to perform invoice label detection using OpenCV, a popular computer vision library. We‘ll cover the core concepts and techniques including image preprocessing, template matching, coordinate extraction, and text recognition. By the end, you‘ll have a solid understanding of how to implement an invoice label detection system using Python and OpenCV. Let‘s dive in!

Invoice Label Detection Architecture

At a high level, an invoice label detection system involves the following key steps:

1. Preprocess the input invoice image to clean it up and prepare it for analysis
2. Determine which template the invoice matches out of a set of known templates
3. Extract the coordinates of the desired labels from the matching template
4. Perform text detection and recognition at those coordinates on the input invoice
5. Output the recognized label text

Here‘s a diagram illustrating this basic architecture:

[Architecture Diagram]

We‘ll go through each of these steps in detail, but first let‘s discuss the important preprocessing required to get the invoice image ready for label detection.

Image Preprocessing Techniques

Invoice images in the wild are often messy – they can be skewed, noisy, low contrast, etc. Proper preprocessing is critical to successful label detection. Some key preprocessing techniques include:

• Binarization – Convert the color/grayscale image to pure black and white pixels which can help improve text detection. Adaptive thresholding methods like Gaussian thresholding are often used.

• Noise filtering – Remove noise artifacts like speckles or blobs that can throw off text detection. Median filtering and bilateral filtering are common denoising methods.

• Skew correction – Detect and correct any skew in the image so that text lines are perfectly horizontal. This can be done by calculating the skew angle and performing a rotational transformation.

• Contour detection – Identify the main content area of the invoice and crop out any background or borders. Finding the largest contour in the image usually accomplishes this.

Here‘s some sample code showing how to perform these preprocessing steps in OpenCV:

import cv2
import numpy as np
from skimage import io
from skimage.filters import threshold_local
from skimage.transform import rotate

# Load image
image = cv2.imread(‘invoice.png‘)

# Convert to grayscale 
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Binarization
T = threshold_local(gray, 11, offset = 10, method = "gaussian")
binary = (gray > T).astype("uint8") * 255

# Noise removal
filtered = cv2.bilateralFilter(binary, 9, 75, 75)

# Skew correction
thresh = cv2.threshold(filtered, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1] 
coords = np.column_stack(np.where(thresh > 0))
angle = cv2.minAreaRect(coords)[-1]
if angle < -45:
    angle = -(90 + angle)
else:
    angle = -angle
rotated = rotate(filtered, angle, resize=True) * 255
rotated = rotated.astype(np.uint8) 

# Contour detection
contours, hierarchy = cv2.findContours(rotated, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
areas = [cv2.contourArea(c) for c in contours]
max_index = np.argmax(areas)
x,y,w,h = cv2.boundingRect(contours[max_index])
cropped = rotated[y:y+h, x:x+w]

After applying these preprocessing steps, the invoice image will be much cleaner and easier to perform label detection on, as shown below:

[Preprocessed Invoice Image]

Detecting the Invoice Template

With the invoice image preprocessed and ready to go, the next step is to determine which template it matches out of a collection of known invoice templates. This is necessary because the coordinates of the labels we want to extract vary across different invoice layouts.

To detect which template the invoice belongs to, we can use image similarity methods to compare the input invoice to each known template. Two common approaches are:

1. Structural similarity (SSIM) – Compares the structural information of the images like luminance, contrast, and structure. Returns a value between -1 and 1 indicating similarity.

2. Feature matching – Detects salient keypoints in both images using algorithms like SIFT or SURF, then matches the features to determine similarity.

In practice, feature-based template matching with ORB keypoints and brute-force matching works well. Here‘s an example of how to implement it:

import cv2

# Load input image and template 
img1 = cv2.imread(‘input_invoice.png‘,0)
img2 = cv2.imread(‘template_invoice.png‘,0)

# Detect keypoints and compute descriptors
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1,None)
kp2, des2 = orb.detectAndCompute(img2,None)

# Match features
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1,des2)
matches = sorted(matches, key = lambda x:x.distance)

# Calculate similarity score
good_matches = matches[:int(len(matches)*0.1)]
similarity = len(good_matches) / len(matches)

We can run template matching between the input invoice and each potential template, then take the one with the highest similarity score as the matching template.

Extracting Label Coordinates from Templates

Now that we‘ve determined the best matching template for the input invoice, we need to get the coordinates of the labels to extract from that template.

To do this, we can annotate each template image beforehand with the bounding boxes of the desired labels. Then we simply load those bounding box coordinates based on which template matched the input invoice.

There are a few ways to get label annotations for the templates:

1. Manually annotate the templates using a labeling tool to draw the bounding boxes

2. Implement a one-time automated annotation process using heuristics based on the template layout (e.g. the invoice # is always in the top right corner). This requires upfront work but then label extraction is automatic.

3. Use a deep learning model trained for spatial layout analysis to predict the label locations. This requires a lot of annotated training data.

For a small number of templates, manual annotation is quick and easy. The label coordinates for each template can be saved in a pandas dataframe or json file for easy lookup.

Here‘s an example of what the template label coordinates might look like:

Template 1:
Invoice Number - [227, 76, 368, 108]
Invoice Date - [735, 75, 882, 107]
Vendor Name - [166, 154, 475, 205]
Total Amount - [807, 1188, 1089, 1240]

Template 2:
Invoice Number - [153, 121, 353, 164]
Invoice Date - [639, 126, 805, 163]  
Vendor Name - [172, 211, 433, 256]
Total Amount - [745, 1104, 978, 1153]

Performing Label Detection on Input Invoices

With the matching invoice template determined and its label coordinates extracted, we‘re finally ready to detect the actual label values on the input invoice.

We can perform text detection and recognition with OpenCV‘s EAST text detector and Tesseract OCR engine. The process looks like this:

1. Load the input invoice image and convert to grayscale
2. For each label:
   • Extract the ROI (region of interest) from the invoice based on the label‘s bounding box coordinates in the matching template
   • Apply the EAST text detector to the ROI to isolate the text region
   • Pass the text region to Tesseract OCR to recognize the text
   • Store the recognized text in a dictionary with the label name as key

Here‘s the code to accomplish this:

import cv2
import pytesseract

def detect_labels(img_path, template):
    img = cv2.imread(img_path)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) 

    label_dict = {}
    for label, coords in template.items():
        x, y, w, h = coords
        roi = gray[y:y+h, x:x+w]

        # Perform EAST text detection
        net = cv2.dnn.readNet("frozen_east_text_detection.pb")
        blob = cv2.dnn.blobFromImage(roi, 1.0, (roi.shape[1], roi.shape[0]), (123.68, 116.78, 103.94), swapRB=True, crop=False)
        net.setInput(blob)
        scores, geometry = net.forward(["feature_fusion/Conv_7/Sigmoid", "feature_fusion/concat_3"])
        boxes = decode_predictions(scores, geometry)

        # Perform text recognition with Tesseract OCR
        for box in boxes:
            x, y, w, h = box
            cropped_roi = roi[y:y+h, x:x+w]
            text = pytesseract.image_to_string(cropped_roi)
            label_dict[label] = text

    return label_dict

Running this on our input invoice image with the label coordinates from the matching template will produce a dictionary containing the recognized text for each label:

{
  ‘Invoice Number‘: ‘INV-12345‘, 
  ‘Invoice Date‘: ‘01/15/2022‘,
  ‘Vendor Name‘: ‘Acme Inc.‘,
  ‘Total Amount‘: ‘$4,567.89‘
}

And with that, we‘ve successfully extracted the key fields from our invoice! There are certainly additional improvements we could make, like fuzzy string matching to correct OCR errors or using multiple OCR engines and ensembling the results, but this covers the basic approach.

Conclusion

To recap, the key steps for invoice label detection using OpenCV are:

1. Preprocess the invoice image using techniques like binarization, noise removal, deskewing, and contour detection
2. Determine the matching template for the invoice using image similarity methods like feature matching
3. Extract the label coordinates for the matching template from an annotation file
4. Perform text detection and recognition on the input invoice image using those label coordinates
5. Output the recognized text for each label in a structured format

The core concepts we covered were:

• Image preprocessing for cleaning up the invoice before analysis
• Template matching to find the invoice‘s layout
• Extracting text regions based on label coordinates
• Text detection with the EAST detector to isolate text areas
• Text recognition with Tesseract OCR to convert images to strings

I hope this guide gave you a good understanding of how to approach invoice label detection using computer vision and OCR. The techniques we discussed provide a solid foundation for building a reliable and efficient invoice processing system. Let me know if you have any other questions!