Learn How To Do Real-Time Background Replacement using OpenCV and CVzone

Real-time background replacement is a powerful computer vision technique that allows you to dynamically change the background of a live video feed. Whether you want to replace a cluttered background with a clean virtual background for video calls, or create immersive AR experiences by superimposing subjects on different scenes, background replacement has a wide range of applications.

In this tutorial, we‘ll learn how to implement real-time background replacement using two awesome open-source libraries: OpenCV and CVzone. By the end, you‘ll be able to run the Python code yourself and swap out your own background with just a few lines of code!

But before we dive into the code, let‘s first get a better understanding of the tools we‘ll be using and why background replacement is so useful.

What are OpenCV and CVzone?

OpenCV (Open Source Computer Vision Library) is a popular open-source library for computer vision and machine learning. Originally developed by Intel, it provides a comprehensive set of optimized algorithms and functions for image and video processing. OpenCV is widely used for tasks like facial recognition, object detection, motion tracking, and more.

While OpenCV provides the core computer vision infrastructure, CVzone is a higher-level library that makes it easy to quickly build complex AI and computer vision features. Under the hood, CVzone uses OpenCV and MediaPipe, but abstracts a lot of the low-level details to provide convenient modules for things like face detection, hand tracking, pose estimation, and more.

Together, OpenCV and CVzone make a powerful combination that allows us to easily implement advanced computer vision applications like real-time background replacement. The best part is both libraries are free, open-source, and well-documented, lowering the barrier to entry.

Why is real-time background replacement useful?

There are many situations where we may want to modify or replace the background of a live video:

• Removing distractions: If you‘re on a video call or recording a presentation, a cluttered or messy background can be distracting to viewers. Background replacement allows you to remove those distractions and keep the focus on the subject.

• Maintaining privacy: Sometimes the background may contain sensitive or personal information you don‘t want to share. Replacing the background protects your privacy.

• Creating professional looking videos: A clean, simple background can make videos look much more professional and polished compared to a busy or uncontrolled background.

• Compositing and special effects: Background replacement is also the basis for a lot of cool visual effects. You can superimpose a subject on any virtual background to create the illusion they are somewhere else. This is often used in movies and gaming for green screen compositing.

• Enabling new experiences: More cutting edge use cases include things like AR try-on where users can see how clothes look on them in different settings, or virtual tours and real estate previews that show how spaces look with different furnishings and decor.

As you can see, background replacement is a handy technique to have in your computer vision toolbelt. Let‘s look at some methods for removing and replacing backgrounds.

Techniques for background replacement

There are several different ways to remove the background from an image or video:

1. Chroma keying (green screen): Probably the most well known method, chroma keying involves filming the subject in front of a single color background (often bright green) which can then be removed based on color thresholding. Requires a physical green screen setup.

2. Image matting and segmentation: These algorithms try to estimate an alpha matte (transparency mask) that indicates the foreground vs background pixels. They segment the image into different regions without requiring a physical green screen. Examples include U2-Net, MODNet, BGMv2.

3. Depth-based segmentation: Depth-sensing cameras like the Kinect and iPhone TrueDepth camera provide per-pixel depth which can be used to separate closer foreground from distant background. Limited to special depth-capable cameras though.

4. Background subtraction: If the camera is stationary, you can model the background over time and detect moving foreground objects by comparing the current frame to the background model. Doesn‘t work for moving cameras/backgrounds though.

5. Semantic segmentation: This technique applies machine learning to classify each pixel into a predefined set of classes like person, dog, sofa, wall, etc. By grouping pixels into different classes, you can identify and extract what you want as the foreground.

In this tutorial, we‘ll be using the SelfiSegmentation module provided by CVzone which internally uses a deep learning model to segment persons from the background. It‘s based on MediaPipe‘s
Selfie Segmentation model.

The advantage of using SelfiSegmentation is that it works pretty well out of the box without needing a green screen, runs in real-time, and is not constrained to stationary cameras. The tradeoff is it only segments person vs background as opposed to more granular segmentation.

Now that we understand the why and how of background replacement, let‘s get our hands dirty with some code and see it in action!

Implementing background replacement with OpenCV and CVzone

Step 0: Make sure you have a working webcam

This tutorial assumes you have a functioning webcam connected to your computer. Most laptops have a built-in webcam, but you may need to install drivers for an external USB webcam.

To test that your webcam is working with OpenCV, run the following code:

import cv2

cap = cv2.VideoCapture(0)
cap.set(3, 640)
cap.set(4, 480) 

while True:
    success, img = cap.read()
    cv2.imshow("Image", img)

    if cv2.waitKey(1) & 0xFF == ord(‘q‘):
        break

If everything is set up correctly, a window should pop up showing the live feed from your webcam! Press ‘q‘ to exit the loop and close the window.

The cap.set(3, 640) and cap.set(4, 480) lines set the width and height of the capture frame to 640×480 pixels. We‘ll be using this resolution throughout the tutorial.

Step 1: Install dependencies

First, let‘s install the required Python packages. You‘ll need OpenCV (opencv-python) for core video capture and processing, CVzone (cvzone) for the SelfiSegmentation module, and MediaPipe (mediapipe) which is a dependency for CVzone.

You can install them using pip:

pip install opencv-python
pip install cvzone
pip install mediapipe

Step 2: Set up background images

To replace the background, we need some background images! Create a new folder in your project directory called "BackgroundImages" or similar.

Download some images you want to use as backgrounds and place them in the folder. For this tutorial, let‘s assume you have 5 images named ‘1.jpg‘, ‘2.jpg‘, … ‘5.jpg‘.

One important thing to keep in mind is that the background images should have the same aspect ratio and resolution as the webcam frames, which we set to 640×480. If the background has a different aspect ratio, it will look stretched or squeezed.

To make sure the backgrounds are the right size, we can use OpenCV to resize them in a quick Python script:

import cv2
import os

bg_dir = ‘BackgroundImages‘

for filename in os.listdir(bg_dir):
    if filename.endswith(".jpg"): 
        img_path = os.path.join(bg_dir, filename)
        bg_img = cv2.imread(img_path)
        bg_img = cv2.resize(bg_img, (640, 480))
        cv2.imwrite(img_path, bg_img)
        print(f"Resized {filename} to 640x480")

This script loops through all the .jpg files in the BackgroundImages directory, loads them with cv2.imread(), resizes them to 640×480 with cv2.resize(), and overwrites the original image with the resized version.

After running this, all your background images should be ready!

Step 3: Implement real-time background replacement

Now for the fun part – let‘s write the code to actually perform background replacement on our live webcam feed.

Here‘s the full code, then we‘ll break it down step-by-step:

import cv2
import cvzone
from cvzone.SelfiSegmentationModule import SelfiSegmentation
import os

cap = cv2.VideoCapture(0)
cap.set(3, 640)
cap.set(4, 480)

segmentor = SelfiSegmentation()
fpsReader = cvzone.FPS()

bg_dir = ‘BackgroundImages‘
bg_imgList = os.listdir(bg_dir)
imgList = []
for imgPath in bg_imgList:
    img = cv2.imread(os.path.join(bg_dir, imgPath))
    imgList.append(img)

indexImg = 0

while True:
    success, img = cap.read()
    imgOut = segmentor.removeBG(img, imgList[indexImg], threshold=0.8)

    imgStack = cvzone.stackImages([img, imgOut], 2, 1)
    _, imgStack = fpsReader.update(imgStack)

    cv2.imshow("Image", imgStack)
    key = cv2.waitKey(1)
    if key == ord(‘a‘):
        if indexImg > 0:
            indexImg -= 1
    elif key == ord(‘d‘):
        if indexImg < len(imgList) - 1:
            indexImg += 1
    elif key == ord(‘q‘):
        break

Let‘s go through it section by section:

First, we import the necessary modules – opencv, cvzone, SelfiSegmentation, and os.

Then we create a VideoCapture object to read frames from the default webcam (index 0) and set the width and height to 640×480 pixels.

cap = cv2.VideoCapture(0)
cap.set(3, 640) 
cap.set(4, 480)

Next, we initialize the SelfiSegmentation object which will handle removing the background, and a FPS reader from cvzone to measure performance.

segmentor = SelfiSegmentation()
fpsReader = cvzone.FPS()

We get a list of all the background images in the BackgroundImages directory using os.listdir(), read each image using OpenCV, and append them to a list called imgList. We also keep track of the current background index in indexImg starting from 0.

bg_dir = ‘BackgroundImages‘
bg_imgList = os.listdir(bg_dir)
imgList = []
for imgPath in bg_imgList:
    img = cv2.imread(os.path.join(bg_dir, imgPath))
    imgList.append(img)

indexImg = 0

Now we start an infinite while loop to continuously process frames from the webcam. In each iteration, we:

1. Read a frame from the webcam using cap.read() which gives us a success boolean and the image.

2. Remove the background from the webcam frame and replace it with the background image at the current index indexImg using segmentor.removeBG(). This is where all the magic happens! The threshold parameter controls how aggressively to remove the background – higher values remove more, lower values remove less. 0.8 is a good default.

3. Stack the original and background-replaced frames side-by-side using cvzone.stackImages() so we can easily compare them. We also overlay the current FPS on the stacked image using fpsReader.update().

4. Display the stacked image in a window using cv2.imshow().

5. Check for keyboard input using cv2.waitKey(1). If the user presses:

   • ‘a‘: move to the previous background image by decrementing indexImg, making sure not to go below 0
   • ‘d‘: move to the next background image by incrementing indexImg, making sure not to exeed the number of images
   • ‘q‘: exit the loop and end the program

while True:
    success, img = cap.read()
    imgOut = segmentor.removeBG(img, imgList[indexImg], threshold=0.8)

    imgStack = cvzone.stackImages([img, imgOut], 2, 1)
    _, imgStack = fpsReader.update(imgStack)

    cv2.imshow("Image", imgStack)
    key = cv2.waitKey(1)
    if key == ord(‘a‘):
        if indexImg > 0:
            indexImg -= 1
    elif key == ord(‘d‘):
        if indexImg < len(imgList) - 1:
            indexImg += 1
    elif key == ord(‘q‘):
        break

And that‘s it! When you run this Python script, you should see a window pop up showing your webcam feed on the left, and the background-replaced version on the right. The current FPS will be displayed at the top. You can press ‘a‘ and ‘d‘ to cycle through the different background images, and ‘q‘ to quit.

Recap and next steps

In this tutorial, we learned how to use OpenCV and the SelfiSegmentation module from CVzone to implement real-time background replacement on a live webcam feed.

The key steps were:

1. Setting up our development environment and installing dependencies
2. Preparing a set of background images with the right resolution
3. Initializing the VideoCapture, SelfiSegmentation, and FPS reader objects
4. Processing the webcam frames in a loop by removing the background and replacing it with a selected background image
5. Displaying the results and listening for user input to change backgrounds or exit

To take this project further, you could try:

• Using different background video instead of static images
• Adding more user control over background selection, threshold value, etc.
• Overlaying graphics, text, or 3D objects on the background-replaced feed to create AR effects
• Streaming the background-replaced video over the network
• Optimizing performance for higher resolutions and framerates

There are tons of creative possibilities! The fundamentals we learned today can be extended in many directions.

Hopefully this tutorial gave you a practical introduction to background replacement with OpenCV and CVzone. The complete code is available on GitHub here: https://github.com/yourusername/background-replacement-demo

Feel free to use it as a starting point for your own projects. Happy hacking!