Detecting 3D Hand Landmarks in Images with MediaPipe
Hand tracking is an exciting and rapidly advancing area of computer vision with a wide range of applications across human-computer interaction, augmented reality, robotics, and more. The ability to accurately detect and track the fine movements of hands and fingers in real-time using standard RGB cameras opens up new possibilities for gesture control, sign language recognition, and intuitive human-machine interfaces.
In June 2020, Google released MediaPipe Hands, a high-fidelity hand and finger tracking solution that uses machine learning (ML) to infer 21 3D landmarks of a hand from a single frame. MediaPipe Hands is cross-platform, runs in real-time on mobile devices and desktops, and is available as open source in the MediaPipe framework.
In this article, we‘ll take a deep technical dive into how MediaPipe Hands works under the hood and walk through a step-by-step tutorial on using it to detect hand landmarks in images with Python. As an AI/ML expert and practitioner, I‘ll share insights into the model architecture, datasets, performance, limitations, and future directions. Whether you‘re an experienced ML engineer or just curious about the cutting edge of hand tracking technology, read on to learn more.
How MediaPipe Hands Works
At a high level, MediaPipe Hands takes an RGB image as input, and outputs the predicted 3D locations of 21 hand landmarks (joints and fingertips). The ML pipeline consists of three main stages:
-
Palm detection: A palm detector model is run on the full input image to detect initial hand regions. This model is a single-shot detector implemented as a convolutional neural network (CNN). It predicts hand bounding boxes and coarse palm landmarks.
-
Hand landmark: For each detected hand region, a hand landmark model is run to predict 21 3D hand keypoints. This model is also a CNN and is trained end-to-end on ~30K real-world images with 3D ground truth annotations. The output is a vector of 63 values representing the (x,y,z) coordinates of each landmark in a normalized coordinate space.
-
Filtering and association: Finally, the predicted hand landmarks are filtered and processed to remove jitter, handle occlusions, and associate hands across consecutive frames in a video stream.
One of the key innovations of MediaPipe Hands is the use of a multi-stage pipeline that progressively focuses on regions of interest. In the first stage, the palm detector generates a coarse hand crop that is then fed to the hand landmark network. This attention mechanism helps improve accuracy (by providing a high-resolution crop for landmark prediction) while keeping computation cost low (by avoiding running the expensive landmark model on the full image).
Another notable feature is the use of 3D hand pose prediction. Whereas many earlier hand tracking methods only predict 2D landmarks, MediaPipe Hands predicts full 3D coordinates including relative depth. This is made possible by training on a large-scale dataset of real hand images with ground truth 3D annotations collected via a novel multi-camera capture setup.
The 3D hand landmark model itself uses several architectural innovations including a CoordNet-based backbone, a 3D skeleton generator for regularizing the predicted landmarks, and a 2.5D heatmap loss for improved keypoint localization [1]. It achieves state-of-the-art accuracy on the public InterHand2.6M 3D hand pose estimation benchmark [2]:
| Method | MPJPE (mm) |
|---|---|
| MediaPipe Hands | 13.27 |
| Zhou et al. [3] | 16.58 |
| Boukhayma et al. [4] | 18.62 |
Table 1. Comparison of 3D hand pose estimation accuracy on the InterHand2.6M dataset. MPJPE = mean per-joint position error in millimeters (lower is better).
In terms of runtime performance, MediaPipe Hands can run at 30 frames per second (FPS) on a high-end desktop GPU and 10 FPS on a mid-tier mobile phone. The model size is ~10MB.
While MediaPipe Hands is quite robust, it does have some limitations. The model is trained primarily on images of individual hands, so it may struggle in scenes with multiple overlapping or interacting hands. The landmark prediction also degrades when large parts of the hand are occluded or outside the image frame. Furthermore, the model exhibits some biases – for example, it tends to perform worse on hands with darker skin tones, possibly due to imbalances in the training data [5].
Coding Tutorial
Now let‘s walk through using MediaPipe Hands in Python to detect hand landmarks in a static image. The full code is available in this Colab notebook.
Step 1: Install MediaPipe and OpenCV.
!pip install mediapipe opencv-pythonStep 2: Import the necessary modules.
import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.handsStep 3: Load the input image.
IMAGE_FILES = [‘hand.jpg‘]
image = cv2.imread(IMAGE_FILES[0])Step 4: Create a MediaPipe Hands object.
with mp_hands.Hands(
static_image_mode=True,
max_num_hands=2,
min_detection_confidence=0.5) as hands:The static_image_mode flag tells the hand tracker that we‘ll be passing in a static image rather than a video stream. The max_num_hands parameter sets the maximum number of hands to detect in the image. And min_detection_confidence sets the minimum confidence value ([0,1]) from the palm detection model for the detection to be considered successful.
Step 5: Convert the BGR image to RGB and process it with MediaPipe Hands.
results = hands.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))Step 6: Print handedness (left or right hand) and draw the hand landmarks on the image.
print(‘Handedness:‘, results.multi_handedness)
if not results.multi_hand_landmarks:
print("No hands detected.")
else:
annotated_image = image.copy()
for hand_landmarks in results.multi_hand_landmarks:
print(‘hand_landmarks:‘, hand_landmarks)
mp_drawing.draw_landmarks(
annotated_image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
cv2.imwrite(‘/tmp/annotated_image.png‘, annotated_image)Here‘s an example of what the output looks like:
Fig 1. Example output of MediaPipe Hands. 21 hand landmarks are detected and plotted on the input image.
Understanding the Hand Landmark Model
Now let‘s take a closer look at what the hand landmark model is actually predicting. The 21 hand landmarks are defined as follows:
Fig 2. The 21 hand landmarks detected by MediaPipe Hands. (Image source: MediaPipe documentation)
The landmark indices are:
- 0-4: Thumb (CMC, MCP, IP, TIP)
- 5-8: Index (MCP, PIP, DIP, TIP)
- 9-12: Middle (MCP, PIP, DIP, TIP)
- 13-16: Ring (MCP, PIP, DIP, TIP)
- 17-20: Pinky (MCP, PIP, DIP, TIP)
Each landmark is a specific, anatomically-defined point of the hand skeleton. The 3D coordinates of each landmark are represented in a normalized coordinate space relative to the hand. The model learns a kinematic skeleton structure representing the bone lengths and joint angles of the hand, which helps constrain the predicted landmarks to biomechanically plausible configurations.
The x and y coordinates are normalized to [0.0, 1.0] by the image width and height respectively. The z coordinate represents the relative depth, with values in [-1.0, 1.0] where a smaller value indicates the landmark is closer to the camera. To get the actual 3D coordinates in millimeters, you need to multiply by the intrinsic camera matrix parameters. However, the relative z values are still useful for many gesture recognition tasks.
Internally, the hand landmark model uses a 2.5D heatmap representation to localize the keypoints in image space [1]. At each heatmap location, it produces a 3D vector representing the probability of the landmark being present at that location and its corresponding depth value. This intermediate representation helps the model handle self-occlusions and achieve sub-pixel accuracy.
Ethics and Fairness Considerations
As with any ML system, it‘s important to consider potential ethical issues and biases. Hand tracking has many beneficial use cases, but could also be used for malicious purposes like unauthorized surveillance or stealing passwords input via gestures.
There are also open questions about demographic and geographic fairness. As mentioned earlier, MediaPipe Hands appears to exhibit lower accuracy on darker skin tones, likely due to underrepresentation in the training data. This performance gap could lead to disparate experiences for end users if not properly accounted for. Google has taken some steps to address this, like expanding the training datasets and using more diverse data augmentation, but more work is needed.
Even with demographically balanced datasets, there may be cultural differences in how people perform gestures that could affect model accuracy. Tracking performance could also vary for users with motor impairments or assistive devices. Careful user studies and disaggregated evaluations across different subpopulations are necessary to characterize and mitigate these issues.
Future Directions
3D hand tracking is a rapidly advancing field with many interesting research directions. Some areas for future work on MediaPipe Hands include:
- Increasing the number of hand landmarks for finer-grained gesture recognition
- Improving robustness to occlusions, motion blur, and low resolution
- Enabling real-time hand-hand and hand-object interaction
- Reducing model size and latency for resource-constrained edge devices
- Federated learning approaches to improve privacy and fairness
- Exploring new applications in AR/VR, robotics, sign language, etc.
Google is actively developing MediaPipe Hands and has already released several updates since the initial launch, like support for Android, iOS, and web (via WebAssembly). They are also working on tracking solutions for other body parts like Holistic (face, hands, pose) and Face Mesh.
It will be exciting to see how hand tracking technology evolves in the coming years. With the increasing accessibility of ML frameworks like MediaPipe, more developers are empowered to build hand tracking into their apps and push the technology in creative new directions. There is still much work to be done to improve accuracy, efficiency, fairness, and robustness, but the potential for positive real-world impact is immense.
Conclusion
In this article, we took a deep dive into MediaPipe Hands, a state-of-the-art ML solution for 3D hand tracking. We examined how the model works, walked through a code tutorial, and discussed key technical concepts. We also explored important considerations around ethics and fairness, and outlined some exciting areas for future research.
3D hand tracking is a powerful tool with many promising applications. It enables computers to understand one of the most natural and expressive ways humans communicate. As the technology matures, we can expect intuitive hand-based interfaces to become increasingly common in domains like AR/VR, robotics, accessibility, and beyond.
At the same time, it‘s critical that we develop these systems thoughtfully and responsibly, with attention to potential negative consequences. By investing in approaches to make hand tracking more accurate, robust, and inclusive, we can work towards a future where this technology benefits everyone equitably.
To learn more about MediaPipe Hands, check out the official documentation, or try it yourself in this Colab notebook. You can also explore other MediaPipe solutions like Holistic and Face Mesh.
What hand tracking applications are you excited about? How do you think this technology will evolve in the coming years? Let me know in the comments!
