Build Your Own Optical Character Recognition System with Google Tesseract and OpenCV

Introduction to OCR

Optical character recognition, or OCR, is the process of converting images of text into machine-readable text. OCR has a wide range of applications, from digitizing printed documents and books, to extracting text from natural scene images for indexing and search.

Some common real-world uses of OCR include:

• Scanning printed documents and converting them to digital text files
• Extracting text from images for indexing and searching (e.g. Google Image Search)
• Recognizing text in natural scene images (e.g. reading street signs or storefronts)
• Processing forms and surveys (e.g. grading multiple choice bubble sheet tests)
• Digitizing handwritten notes and historical documents
• Helping visually impaired users read text in images

At a high level, OCR involves two main stages: text detection (locating regions of text in an image) and text recognition (converting the image of text into actual text). Traditionally, OCR systems used computer vision and image processing techniques for text detection and template matching or feature extraction plus machine learning for text recognition.

However, in recent years deep learning has dramatically improved the accuracy of both text detection and recognition. Many modern OCR systems now leverage deep neural networks trained on large datasets of annotated text images.

Despite the advancements from deep learning, building an accurate and robust OCR system for real-world applications remains a significant challenge. The main difficulties arise from the diversity of text appearance in natural scenes – varying fonts, colors, scales, orientations, lighting conditions, image noise and blur, occlusion, and complex backgrounds.

Successfully extracting text also often requires careful preprocessing of images to enhance contrast, remove noise, correct perspective distortion, and so on. Post-processing is also often needed to correct errors in the raw OCR output and reconstruct the original document structure.

In this post, we‘ll walk through how to build your own OCR system using two popular open source tools – Google‘s Tesseract OCR engine and the OpenCV computer vision library. We‘ll cover the fundamentals of text detection and recognition, and share code examples of a complete OCR pipeline. We‘ll also discuss alternative approaches using deep learning models and provide an overview of the state-of-the-art in OCR as of 2024.

Tesseract OCR Engine

Tesseract is an open source OCR engine originally developed by HP in the 1980s. In 2006, HP released Tesseract as open source and since then it has been sponsored by Google. Tesseract started as a traditional feature extraction and machine learning based OCR system, but since version 4.0 released in 2018, it has added a new OCR engine based on long short-term memory (LSTM) neural networks.

Tesseract supports over 100 languages and can perform OCR on both printed documents and natural scene images. It‘s widely used and deployed in many commercial and open source applications.

Out of the box, Tesseract performs both text detection and text recognition. But it works best on high-quality images with clean, well-separated text lines. On complex natural scene images, the accuracy of Tesseract‘s default text detection is often quite poor.

Let‘s look at a quick example of using Tesseract. First, you‘ll need to install the Tesseract executable on your system. Refer to the official Tesseract documentation for installation instructions for your platform.

You can run Tesseract via the command line:

tesseract input.png output -l eng

This will read the input image input.png, perform OCR, and write the recognized text to output.txt, using the English language model specified by -l eng. Tesseract has a variety of command-line options to control the OCR engine mode, page segmentation mode, and other settings. Run tesseract --help for the full list of options.

For using Tesseract in Python, you can install the Pytesseract package, which provides a simple Python wrapper for the Tesseract executable:

pip install pytesseract

Here‘s an example usage of Pytesseract:

from PIL import Image
import pytesseract

img = Image.open("input.png")
text = pytesseract.image_to_string(img, lang="eng")
print(text)

This script reads an input image using the Python Imaging Library (PIL), performs OCR using Tesseract, and prints out the recognized text. The lang parameter specifies the language to use (English in this case).

While Tesseract works well for many OCR tasks out of the box, getting the best accuracy often requires some amount of image preprocessing and post-processing, especially for natural scene images. Some common techniques include:

• Binarization: Converting the image to pure black-and-white by applying a threshold
• Noise removal: Removing small specks and blobs via morphological operations or filters
• Skew correction: Correcting any rotation or perspective distortion of the text
• Layout analysis: Detecting and separating blocks of text, images and other regions
• Language model rescoring: Refining the OCR output by constraining it to a dictionary or language model
• Error correction: Automatically correcting common OCR errors via heuristics or machine learning

We‘ll see examples of some of these techniques in the next section on using OpenCV for text detection and image preprocessing.

Text Detection with OpenCV

OpenCV is a popular open source computer vision library that provides a wide range of image processing and analysis functions. For OCR, we can use OpenCV to detect regions of text in an image, and extract those regions as separate crops to pass to Tesseract for final text recognition. This divide-and-conquer approach often gives much better accuracy than running Tesseract on the full image.

The basic steps for text detection with OpenCV are:

1. Preprocess the image (convert to grayscale, remove noise, apply thresholding, etc.)
2. Find contours in the image
3. Filter and group the contours into candidate text regions
4. Perform additional filtering and layout analysis to refine the text regions
5. Crop the final text regions from the original image

Here‘s an example Python script that demonstrates these steps:

import cv2
import pytesseract
from pytesseract import Output

def detect_text(img):
    # Preprocess image
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray = cv2.medianBlur(gray, 3)
    thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

    # Find contours
    contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0]

    # Filter contours
    min_area = 100
    max_area = 1000000
    contours = [c for c in contours if min_area < cv2.contourArea(c) < max_area]

    # Sort contours from top to bottom
    contours.sort(key=lambda c: cv2.boundingRect(c)[1])

    # Group contours into lines
    max_vert_dist = 10
    lines = [[]]
    for c in contours:
        x, y, w, h = cv2.boundingRect(c)
        if len(lines[-1]) > 0:
            prev_x, prev_y, prev_w, prev_h = cv2.boundingRect(lines[-1][-1]) 
            if y - (prev_y + prev_h) > max_vert_dist:
                lines.append([])
        lines[-1].append(c)

    # Merge contours in each line
    regions = []
    for line in lines:
        x = min([cv2.boundingRect(c)[0] for c in line]) 
        y = min([cv2.boundingRect(c)[1] for c in line])
        w = max([(cv2.boundingRect(c)[0] + cv2.boundingRect(c)[2]) for c in line]) - x
        h = max([(cv2.boundingRect(c)[1] + cv2.boundingRect(c)[3]) for c in line]) - y
        regions.append((x, y, w, h))

    return regions

img = cv2.imread("input.png")
regions = detect_text(img)

for i, (x, y, w, h) in enumerate(regions):
    crop = img[y:y+h, x:x+w]
    text = pytesseract.image_to_string(crop, lang="eng")
    print(f"Detected text {i}: {text}")

This script performs the following steps:

1. Converts the input image to grayscale, applies median blurring to reduce noise, and performs binary thresholding using Otsu‘s method to get a clean black-and-white image.

2. Finds contours in the thresholded image. Contours are basically connected regions of white pixels. We use external contour retrieval mode to only get the outermost contours.

3. Filters the contours by area to remove very small or very large regions that are unlikely to be text.

4. Sorts the remaining contours from top to bottom based on their bounding rectangle coordinates.

5. Groups the sorted contours into lines based on vertical distance. Contours that are close together vertically are grouped into the same line.

6. Merges the contours in each line into a single bounding rectangle.

7. Crops the merged text region rectangles from the original image and runs each crop through Tesseract to perform final text recognition.

This is just one example of a simple text detection pipeline using OpenCV. In practice, you may need to experiment with different preprocessing techniques, contour filtering criteria, and layout analysis approaches to get the best results for your specific use case.

Alternative Approaches with Deep Learning

While traditional computer vision techniques like contour analysis can work well for text detection in some cases, they often struggle with more complex natural scene images. In recent years, deep learning models have achieved state-of-the-art results on both text detection and recognition.

Some popular deep learning based text detection models include:

• EAST (Efficient and Accurate Scene Text): A fully convolutional neural network that performs text detection in a single pass. Achieves high accuracy and real-time speeds.

• CRAFT (Character Region Awareness for Text Detection): A deep neural network that detects individual characters and links them together into text regions. Robust to curved, rotated, and irregularly shaped text.

• TextBoxes / TextBoxes++: A set of convolutional neural networks for text detection based on the SSD (Single Shot MultiBox Detector) object detection framework.

For text recognition, some state-of-the-art deep learning models include:

• CRNN (Convolutional Recurrent Neural Network): Combines convolutional layers for feature extraction with recurrent layers for sequence modeling. Performs unconstrained text recognition without character segmentation.

• Attention OCR: A neural network that uses attention mechanisms to focus on different parts of the input image when predicting each character. Achieves state-of-the-art accuracy on several benchmarks.

• Transformer-based models: Recent models based on transformer architectures (like the popular BERT model for natural language processing) have pushed the boundaries of text recognition accuracy even further.

Deep learning models are able to learn robust feature representations directly from data, which allows them to better handle the wide variability of real-world text images. The downside is they typically require a large amount of annotated training data and significant computational resources to train.

In practice, the choice of whether to use a traditional pipeline with Tesseract or a deep learning model depends on your specific requirements for accuracy, speed, and training data availability. For many applications, a hybrid approach that combines deep learning for text detection with Tesseract for text recognition can offer a good balance of accuracy and ease of use.

Building Robust OCR Systems

Building an OCR system that works reliably in the real world requires more than just stringing together off-the-shelf components. Some key considerations include:

• Image quality control: Ensuring input images are of sufficient resolution and quality for accurate OCR. This may require manual or automated quality checks and feedback to users.

• Preprocessing and normalization: Developing robust preprocessing pipelines to handle the variability of real-world images. This includes techniques like binarization, noise removal, skew correction, and layout analysis.

• Model selection and optimization: Choosing the right detection and recognition models for your use case and optimizing them for accuracy and speed. This may require experimenting with different architectures, training datasets, and hyperparameters.

• Postprocessing and error correction: Implementing heuristics or machine learning models to automatically correct common OCR errors and postprocess the raw OCR output into a structured format.

• Performance monitoring and continuous improvement: Deploying monitoring and analytics to track the performance of your OCR system in production and identify areas for improvement. Continuously collecting and annotating real-world data to improve model accuracy over time.

Conclusion and Future Directions

OCR is a complex and rapidly evolving field that combines computer vision, machine learning, and natural language processing. While OCR accuracy has improved dramatically in recent years thanks to deep learning, building truly robust and reliable OCR systems for real-world applications remains a significant challenge.

Some key open research problems and future directions for OCR include:

• Few-shot and unsupervised learning approaches to reduce the amount of annotated training data required for building custom OCR models.

• Domain adaptation and transfer learning techniques to improve the performance of OCR models on out-of-distribution data (e.g. historical documents, handwriting, specialized domains).

• Approaches for preserving the layout and structure of the original documents (e.g. identifying section headings, bulleted lists, tables) rather than just outputting unstructured text.

• Multimodal approaches that combine visual and textual information for OCR in the wild (e.g. leveraging surrounding visual context to aid text recognition).

• Tighter integration of OCR with downstream natural language processing tasks like named entity recognition, relation extraction, and question answering.

As the volume of unstructured visual data continues to grow, OCR will play an increasingly crucial role in making this information accessible and useful. Alongside rapid advances in computer vision and deep learning, we can expect to see OCR systems become ever more accurate, efficient, and widely deployed in the years ahead.