CRAFT: State-of-the-Art Text Detection via Character-Region Awareness

Text detection is a fundamental task in computer vision with a wide range of applications, from document digitization to scene understanding. The goal is to localize all text instances in an image, typically by drawing bounding boxes around each word or text line. Accurate text detection is a key prerequisite for further text recognition and information extraction.

In recent years, deep learning has dramatically advanced the state-of-the-art in text detection, with a variety of novel approaches proposed. One of the most prominent and widely adopted methods is CRAFT (Character-Region Awareness For Text Detection), introduced by Baek et al. from Clova AI Research [1]. CRAFT is a highly accurate, robust, and efficient text detector that has achieved top results on multiple benchmarks.

In this post, we‘ll dive deep into the CRAFT model from an AI/ML expert perspective. We‘ll examine its architecture, training process, and key results – with the goal of understanding how and why it works so well. We‘ll also walk through a code example of using CRAFT, discuss its real-world applications, and consider future research directions. Let‘s get started!

Characteristics of Scene Text

To appreciate the challenges of text detection and the innovations of CRAFT, it‘s helpful to first consider the unique characteristics of scene text images:

1. Diversity of appearance: Unlike scanned documents with consistent fonts and layouts, text in natural scenes can have widely varying colors, scales, fonts, and styles.

2. Irregular layouts: Scene text is often arranged in irregular positions, orientations, and shapes – such as curved or perspectively distorted text.

3. Complex backgrounds: Text in the wild is embedded in complex natural backgrounds and can be partially occluded or have low contrast.

These challenges make traditional OCR approaches designed for documents ineffective for scene text. Instead, modern scene text detectors aim to be robust to these unconstrained conditions by learning more flexible representations.

The CRAFT model in particular addresses these challenges by 1) detecting individual characters to handle arbitrary text shapes, 2) using multi-scale feature maps for robustness to varying sizes, and 3) training on a combination of synthetic and real data to learn more robust features. We‘ll see how these key ideas come together in the model architecture.

Model Architecture

At a high level, the CRAFT model is a fully convolutional neural network that simultaneously predicts character-level segmentation maps and character affinity maps, which are then combined to produce word-level bounding boxes. Let‘s break this down step-by-step.

Figure 1: CRAFT model architecture (Source: [1])

Backbone

The backbone of CRAFT is the VGG-16 network [2] pre-trained on ImageNet. The fully connected layers are removed to make the network fully convolutional, allowing it to process images of arbitrary size. The backbone extracts generic low- and mid-level visual features from the image.

U-Net Decoder

On top of the VGG backbone, CRAFT adds a U-Net-style [3] decoder that upsamples and combines multi-scale feature maps. The decoder helps capture both local and global contexts and improves robustness to scale variation. It consists of a series of upsampling and concatenation operations that progressively increase the feature map resolution.

Region and Affinity Branches

The output of the U-Net decoder is fed into two parallel branches:

1. The region branch predicts a pixel-wise character segmentation map, where each pixel is classified as text or non-text. This branch has a single 1×1 convolutional layer.

2. The affinity branch predicts a set of affinity maps that encode the relationship between neighboring characters. It has $k$ 3×3 convolutional layers, where $k$ is the number of affinity directions (e.g. top, bottom, left, right). The affinity maps aim to group characters into words.

Both branches output maps with the same spatial dimensions as the input image. The region map has a single channel, while the affinity map has $k$ channels.

The key insight of CRAFT is that by predicting character-level regions and affinities instead of word-level bounding boxes, the model can handle text with arbitrary shapes and layouts. The affinity maps also provide a natural way to group characters into words without heuristic post-processing.

Loss Function

During training, the CRAFT model is optimized with a combination of two loss functions:

$$L = \lambda_r L_r + \lambda_a L_a$$

where $L_r$ and $L_a$ are the losses for the region and affinity branches, and $\lambda_r$ and $\lambda_a$ are corresponding weights (hyperparameters).

The region loss $L_r$ is a binary cross-entropy loss that encourages pixels within each character to be classified as text. The ground-truth character regions are generated by applying a 2D Gaussian kernel around each character center.

The affinity loss $L_a$ is a cross-entropy loss that encourages pairs of pixels within the same word to have high affinity scores. The ground-truth affinity maps are generated based on the relative positions of characters within each word.

By jointly optimizing the region and affinity losses, the model learns to accurately segment characters while also grouping them into words.

Synthetic Pre-training

One of the key challenges in training text detection models is the lack of large-scale, fully annotated real-world datasets. To overcome this, the CRAFT authors propose a synthetic pre-training strategy.

First, they generate a large synthetic dataset by rendering text with random fonts, colors, and layouts on various background images. This synthetic data is used to pre-train the CRAFT model, allowing it to learn a good initial representation for text.

The synthetically pre-trained model is then fine-tuned on a smaller dataset of real-world images, such as ICDAR 2013 [4]. This two-stage training process allows the model to both learn robust text features from synthetic data and adapt to the domain of real images.

Inference Pipeline

At test time, the CRAFT model is applied to an input image to produce word-level text detections in three steps:

1. Region segmentation: The input image is resized to a fixed resolution (e.g. 1280×768) and fed into the CRAFT network, which outputs a character region map and a set of affinity maps.

2. Character grouping: The region map is binarized with a threshold (e.g. 0.5) to identify candidate character regions. The affinity maps are then used to group characters into words by computing a score between each pair of characters based on the mean affinity along their boundary.

3. Word bounding box generation: The grouped characters are merged into word-level bounding boxes by taking the minimum and maximum coordinates of each group. No further post-processing or refinement is necessary.

This simple and efficient inference pipeline is made possible by CRAFT‘s character-level detection and affinity-based grouping. It achieves state-of-the-art performance without the need for complex post-processing steps like non-maximum suppression or text line grouping.

Experiments and Results

The CRAFT authors conduct extensive experiments to validate the effectiveness of their approach. Here are some of the key results:

Benchmark Performance

CRAFT achieves state-of-the-art performance on a range of standard text detection benchmarks:

Dataset	F-score
ICDAR 2013	93.1%
ICDAR 2015	91.1%
ICDAR 2017 MLT	81.1%
Total-Text	87.5%

These results demonstrate the strong performance of CRAFT on both regular and irregularly shaped text. In particular, CRAFT outperforms most other methods on datasets like Total-Text that contain highly curved and rotated text instances.

Ablation Study

The authors also conduct an ablation study to validate the importance of each component in the CRAFT architecture. They find that:

• The affinity branch is crucial for grouping characters into words, leading to a 3.5% improvement in F-score on ICDAR 2015.
• The U-Net decoder improves performance by 1.3% by capturing multi-scale context.
• Pre-training on synthetic data improves the F-score by 2.7%, demonstrating the importance of the two-stage training pipeline.

These results confirm that each component of CRAFT contributes meaningfully to its final performance.

Computational Efficiency

In addition to its strong detection accuracy, CRAFT is also computationally efficient. The authors report an inference time of 200ms per image on a single NVIDIA Tesla P40 GPU, which is competitive with other state-of-the-art models.

The fully convolutional architecture of CRAFT allows it to process images of arbitrary size without requiring sliding windows or other inefficient techniques. The simple post-processing pipeline also contributes to its efficiency.

Applications

The ability to accurately detect text in unconstrained environments opens up a wide range of applications across many domains, such as:

• Autonomous driving: Detecting and reading traffic signs, license plates, and street names for navigation and mapping.

• Robotics: Enabling robots to read signs, labels, and other text in the environment for better situational awareness.

• Accessibility: Helping visually impaired users navigate the world by reading out text in their surroundings.

• Document analysis: Extracting text from scanned documents, receipts, and forms for automated data entry and processing.

• Augmented reality: Overlaying translations and other information on top of text in real-world scenes.

By providing a fast and accurate text detection solution, CRAFT helps enable these and many other applications.

Code Example

To demonstrate how to use CRAFT for text detection, let‘s walk through a code example using the official PyTorch implementation from Clova AI Research [5].

First, we need to install the necessary dependencies and download the pre-trained model:

git clone https://github.com/clovaai/CRAFT-pytorch
cd CRAFT-pytorch
pip install -r requirements.txt
wget https://drive.google.com/uc?id=1bupFXqT-VU6Jjeul13XP7yx2Sg5IHr4J -O craft_mlt_25k.pth

Next, we can use the provided test.py script to run inference on an input image:

python test.py --trained_model=craft_mlt_25k.pth --test_folder=./figures

This will apply the CRAFT model to all images in the figures directory and save the results in the result directory.

To use CRAFT in your own code, you can follow this template:

import torch
import craft_utils
import imgproc

# Load the pre-trained CRAFT model
model = craft_utils.load_craftnet_model(‘craft_mlt_25k.pth‘)
model.eval()

# Load an input image
image = imgproc.loadImage(‘input.jpg‘)

# Resize the image to a fixed size
image, _ = imgproc.resize_aspect_ratio(image, 1280, 1280)

# Normalize the image
image = image.astype(‘float32‘) / 255
image = imgproc.normalizeMeanVariance(image)

# Convert to PyTorch tensor and add batch dimension
image = torch.from_numpy(image).permute(2, 0, 1).unsqueeze(0)

# Run inference
with torch.no_grad():
    output = model(image)

# Apply thresholding and grouping to get word-level bounding boxes
text_threshold = 0.7
link_threshold = 0.4
low_text = 0.4
bboxes, polys, _ = craft_utils.getDetBoxes(
    output[0], output[1], text_threshold, link_threshold, low_text
)

# Draw the bounding boxes on the input image
img_with_boxes = imgproc.cvt2HeatmapImg(image.squeeze().numpy())
for box in bboxes:
    img_with_boxes = imgproc.drawPolygon(img_with_boxes, box)

# Save the output image
imgproc.saveImage(‘output.jpg‘, img_with_boxes)

This script loads the pre-trained CRAFT model, preprocesses an input image, runs inference, and draws the predicted word bounding boxes on the image. You can adjust the text_threshold, link_threshold, and low_text parameters to control the sensitivity of the detector.

That‘s it! With just a few lines of code, you can use CRAFT to detect text in your own images. For more advanced usage and fine-tuning, check out the official CRAFT repository [5].

Conclusion and Future Work

In this post, we‘ve taken a deep dive into the CRAFT text detection model from an AI/ML expert perspective. We‘ve seen how CRAFT achieves state-of-the-art performance on a range of benchmarks by combining character-level detection, affinity-based grouping, and a two-stage training pipeline.

The key innovations of CRAFT include:

1. Detecting characters instead of words to handle arbitrary text shapes and layouts
2. Using affinity maps to group characters into words without heuristic post-processing
3. Pre-training on synthetic data to learn robust text features

Through extensive experiments, the authors demonstrate the effectiveness of each component and the computational efficiency of the overall model.

Looking forward, there are still many opportunities for further improving scene text detection models like CRAFT, such as:

• Incorporating contextual information from the surrounding image to handle more challenging cases like partially occluded or low-contrast text
• Exploring more efficient architectures and training strategies to reduce computational cost and latency
• Adapting the model to handle text in multiple languages and scripts

Additionally, future work could explore how to better integrate text detection with downstream tasks like text recognition and scene understanding. For example, a joint detection-recognition model could potentially improve the accuracy of both tasks by sharing features and allowing for end-to-end optimization.

Overall, CRAFT represents an important milestone in scene text detection research, and its open-source release has helped accelerate progress in this field. By understanding its key ideas and innovations, AI/ML practitioners can build on its success to create even more powerful and robust text detection systems.

References

[1] Y. Baek, B. Lee, D. Han, S. Yun, and H. Lee, "Character Region Awareness for Text Detection," in CVPR, 2019.
[2] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," in ICLR, 2015.
[3] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," in MICCAI, 2015.
[4] D. Karatzas et al., "ICDAR 2013 Robust Reading Competition," in ICDAR, 2013.
[5] CRAFT-pytorch, https://github.com/clovaai/CRAFT-pytorch