Enclosed shape extraction from hand-drawn images

Extracting Enclosed Shapes from Hand-Drawn Images: A Comprehensive Guide

Introduction

Hand-drawn images like flowcharts, diagrams, mind maps and technical drawings are commonly used to express ideas and designs in many fields. However, these physical drawings are difficult to edit, share and store compared to digital formats. Converting the hand-drawn images into digital representations enables a wide range of useful applications like:

• Easy modification and version control
• Searchable text within the images
• Automatic generation of digital documents from the drawings
• Analysis and data extraction from diagrams
• Conversion between different diagram formats

In this article, we‘ll walk through how to automatically extract the individual shapes and symbols from a hand-drawn image using computer vision techniques. The ability to isolate the shapes is an important first step for digitization pipelines that eventually convert the shapes into editable objects with recognized text.

We‘ll be using Python and OpenCV to implement the following steps:

1. Binarization – convert input image to black and white
2. Remove text, arrows & noise – filter out small components using connected component analysis
3. Fill in shapes – use morphological operations and flood fill to get solid enclosed regions
4. Extract shapes – detect and crop out the filled in shapes from the image

While advanced deep learning approaches are also viable for this task, the traditional computer vision techniques covered here are still very relevant, especially for resource-constrained applications. The full source code is provided so you can easily experiment with the concepts yourself.

Step 1 – Binarization

The first step is to convert the input image into a binary black and white image, separating the background from the foreground pixels of interest. While this is trivial for born-digital images, real-world handwritten images can be more challenging due to noise and uneven illumination.

Simple binary thresholding applies a fixed cutoff value, setting pixels above the threshold to white and below to black. This works well for evenly lit images with good contrast. For example:

import cv2
img = cv2.imread(‘diagram.jpg‘, 0)
_, binary_img = cv2.threshold(img, 128, 255, cv2.THRESH_BINARY)

The threshold value of 128 assumes pixel intensities range from 0 to 255. You may need to tune it based on your input images.

Adaptive thresholding calculates different threshold values for each small region of the image. This handles cases where the image contrast varies in different areas. OpenCV provides two adaptive methods:

binary_img = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)

binary_img = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 11, 2)

The parameters control the size of the region and the constant subtracted from the mean/weighted mean.

Otsu‘s binarization automatically chooses an optimal threshold value that maximizes the separation between foreground and background intensities. It works well if your image has a clear bimodal distribution of pixel values:

_, binary_img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)

In practice, test out the different binarization approaches on sample images to see which gives the cleanest background-foreground separation for your use case.

Step 2 – Remove Text, Arrows & Noise

The binary image from step 1 will contain all the connected black pixels, which includes the shapes we want but also a lot of small components we want to ignore:

• Text labels
• Arrows and connecting lines
• Small noise regions from binarization

To filter these out, we‘ll apply connected component analysis to extract all the separately connected regions in the image:

numlabels, labels, stats, = cv2.connectedComponentsWithStats(binary_img, 8, cv2.CV_32S)

This gives us the number of unique connected components num_labels, a labels matrix assigning each pixel to a component, and stats about each component like its area, width, height.

We can then remove small components by area like:

MIN_AREA = 100 # components smaller than 100 pixels will be removed
for label in range(1, num_labels):
if stats[label, cv2.CC_STAT_AREA] < MIN_AREA:
binary_img[labels == label] = 0

This sets all pixels belonging to small components to 0 (background). Tune MIN_AREA higher to be more aggressive in removing non-shape components.

We‘re not done yet though, as the remaining larger components may have holes inside them which would interfere with the later flood fill step. To fix these holes, we can apply a morphological closing operation:

kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3))
binary_img = cv2.morphologyEx(binary_img, cv2.MORPH_CLOSE, kernel)

This will fill in small gaps and holes, but still retain the overall shapes. The size of the kernel controls the scale of holes that get filled.

Step 3 – Fill in Enclosed Shapes

Now that we have a pretty clean binary image without unwanted components, we can fill in the enclosed shapes to make them solid.

OpenCV provides the flood fill function for this:

cv2.floodFill(binary_img, mask=None, seedPoint=(0,0), newVal=255)

Starting from the seed point (0,0), this recursively fills in all connected background (0) pixels with white (255), until it hits the black (255) boundary pixels. Applying this to our cleaned binary image will fill all the enclosed shapes white. Make sure your shapes are fully enclosed first or the fill will leak out.

Step 4 – Extract Filled Shapes

The final step is to detect the filled in shapes and extract them by cropping the original image at those locations.

First, we find the external contours of all white regions:

contours, _ = cv2.findContours(binary_img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

Then for each contour, compute its bounding rectangle and crop it out of the original input image:

shapes = [] for contour in contours:
x,y,w,h = cv2.boundingRect(contour)
shape = img[y:y+h, x:x+w] shapes.append(shape)

To get the final shape images with transparent backgrounds, create RGBA images and copy the extracted grayscale shapes into the alpha channel:

shape_rgba = np.zeros((h,w,4), dtype=np.uint8)
shape_rgba[:,:,3] = shape
shapes.append(shape_rgba)

We now have each of the individual shapes extracted as separate images! Here‘s a visualization of the pipeline:

[input image] -> [binary image] -> [removed small components] -> [filled shapes] -> [extracted shape 1] [extracted shape 2] …

These images can then be passed to OCR models to recognize any text inside them, or saved to SVG/HTML/digital design file formats for editing the shapes themselves.

Performance Considerations

The above approach can detect arbitrary shapes but does have some limitations:

• Shapes must be fully enclosed to be filled and extracted properly
• Overlapping or touching shapes will be merged together
• Curved shapes will have a rectangular bounding box, which may include some background pixels
• Slow for large, complex images

Alternatives and Improvements

• Use edge detection like Canny before finding contours for better shape boundary extraction
• Non-max suppression when shapes are inside other shapes
• Separate text vs shape extraction and don‘t fill in shapes, to avoid losing interior detail
• Deep learning object detection models can directly learn to detect shapes without needing the fill operation. More complex to train but can be more robust

Conclusion

Automatically extracting shapes from hand-drawn images is a powerful building block to enable digitizing diagrams, engineering designs, mind maps and more. OpenCV provides accessible functions for the key operations:

• Thresholding for binarization
• Connected component filtering to remove noise
• Flood fill to solidify the shapes
• Contour detection to extract shapes

Pair this with OCR on the extracted shapes to start converting unstructured drawings into structured digital versions. While we covered the core traditional techniques here, recent advances in deep learning are certainly also worth exploring for more end-to-end approaches.

I hope this guide provided a solid foundation to get you started with enclosed shape extraction from images. Let me know if you have any questions!