MIT‘s "X-Ray Vision" AI Can See Through Walls Using Radio Signals

Introduction

A team of researchers at MIT‘s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a neural network that can detect people‘s pose and movement from the other side of a wall. By analyzing radio signals that reflect off human bodies, the AI can create a stick figure representation of a person‘s actions and identity, without requiring any cameras or wearable sensors.

This groundbreaking "RF-Pose" technology, unveiled in 2018, brought the science fiction dream of X-ray vision a step closer to reality. The CSAIL team, led by professor Dina Katabi, says RF-Pose could have many beneficial applications, from monitoring diseases and assisting the elderly, to enhancing virtual reality games and aiding search and rescue operations. At the same time, it raises concerning questions about privacy and surveillance.

In this article, we‘ll take an in-depth look at how RF-Pose works, the challenges the researchers overcame in developing it, the system‘s capabilities and limitations, and its exciting but potentially unsettling implications for the future. As computers gain ever more advanced perception and reasoning abilities, this research offers a glimpse of an AI-enhanced world to come.

How RF-Pose Works

The key to RF-Pose is a neural network that can interpret radio signals bouncing off the human body. Traditional camera-based computer vision looks for visual patterns to detect objects and people. The CSAIL team realized they could apply similar deep learning techniques to analyzing radio frequency (RF) signals.

As lead researcher Mingmin Zhao explains, "By using radio signals, we can see through walls and other occlusions, and track a person‘s movements in a room or building. We use a deep neural network to analyze the radio signal reflected off a person‘s body and then create a stick figure representation of that person and their movements."

Here‘s how it works in more detail:

First, a device transmits low-power radio waves that travel through walls and reflect off any human bodies on the other side. Think of it like the way a bat uses echolocation to "see" its environment and prey.

Next, a receiver captures these reflected radio signals and inputs them into the neural network for processing. The AI analyzes subtle patterns in how the signal changes to determine if a person is present, where they are located, what pose they are in, and how they are moving in real-time.

Finally, the system outputs a simple stick figure avatar on a screen representing the person‘s live pose and movements. Importantly, this is not a detailed image as you would get from a camera, but more of a simplified skeleton model, like a moving stick figure drawing.

The researchers had to develop novel neural network architectures specifically for interpreting the RF signal data. They used an adapted version of a type of deep learning model called a convolutional neural network (CNN), commonly used for image classification tasks.

In this case though, instead of learning visual features like edges and textures, the CNN learns to recognize how different body positions and motions alter the radio signal reflections. It detects key "landmarks" on the body such as head, chest, arms and legs, and then infers overall pose from their relative positions, rather like our brains construct a mental picture of someone‘s pose from seeing how their body parts are arranged.

Training the Neural Network

Of course, the neural network needs to be trained on many examples of paired RF signals and human body poses before it can recognize them in the wild. This training process was one of the key challenges the team had to solve, as Professor Katabi notes:

"There are no datasets that contain radio signals and corresponding stick figure data. So we had to collect our own data, which was a major challenge."

To build their bespoke dataset, the researchers set up a controlled room with both RF transmitters/receivers and standard RGB video cameras. As volunteers performed different actions and poses in the room, the system captured synchronized radio signals and camera footage.

They then used computer vision and deep learning techniques to extract simplified stick figure representations from the regular video frames. This allowed them to create thousands of time-aligned examples pairing raw RF signal input with the "ground truth" body pose labels needed to train the AI model.

Once trained on this unique labeled dataset, the neural network learns the complex function for mapping between radio reflections and 2D human stick figures, without requiring the regular camera images anymore. When deployed in the real-world, it can then infer people‘s live poses from just the RF signals alone as they reflect off the body and pass through obstructions.

Results and Accuracy

So how well does this RF-based neural network actually work at detecting people and poses through walls and occlusions? The CSAIL team rigorously tested their system and reported some impressive results.

In a setup where volunteers moved around in a room behind a wall, RF-Pose successfully detected the presence of a person 100% of the time, and could distinguish between up to 100 different individuals with 83% accuracy based on their style of movement alone. It could also determine a person‘s 2D stick figure pose (i.e. standing, sitting, walking etc.) with 75% accuracy.

When people were fully occluded and not in direct line-of-sight, the pose estimation accuracy dropped to a still respectable 62% for a single person. However, accuracy degraded further to 20% when two people were in the room, due to the complexity of the overlapping radio signal reflections.

The researchers note a few other key limitations of the current system:

• It struggles with pose estimation when people are touching or very close together, as the signals interfere
• It requires specialized hardware to emit and capture the RF signals, so is not compatible with standard WiFi equipment
• The 2D stick figures do not reveal appearance details like faces or clothing textures
• It has only been tested indoors in relatively small room-sized areas, not large buildings or outdoors

Despite these constraints, RF-Pose is still a highly novel and promising approach. Never before has an AI system been able to detect human identity and activity through walls with such accuracy, let alone create a live "skeleton" avatar, all with no cameras or on-body sensors.

Potential Applications

The CSAIL team envisions many beneficial applications for RF-Pose and its successors, such as:

• Monitoring patients with Parkinson‘s, MS, and other mobility-limiting diseases, and alerting caregivers of falls or issues
• Allowing the elderly and disabled to live more independently while discreetly checking on their safety and daily activities
• Enhancing virtual reality games and experiences with real-time body tracking without cumbersome suits or headsets
• Enabling "smart homes" to detect occupants‘ locations and activities and adjust lighting, heating, music etc.
• Helping search and rescue teams locate survivors in rubble or wreckage and assess their condition
• Providing security personnel "X-ray vision" to identify intruders in buildings or behind barriers

As the researchers explained, "We‘re working to turn this technology into an in-home device that can call 911 if it detects that a family member has fallen unconscious. You could also imagine it being used to track the movements of animals in the wild, or even people in assisted-living facilities."

Progress in Computer Vision and AI

RF-Pose is just one example of the rapid advances happening in computer vision and AI perception abilities. In the past decade, deep learning has revolutionized this field, allowing computers to recognize objects, faces, text, speech, and activities with near-human or even superhuman accuracy.

We now have AIs that can drive cars, diagnose diseases from scans, identify bird species, read lips, compose music, write code, and much more. RF-Pose shows how techniques like neural networks are also moving beyond standard camera data to give computers novel modes of sensing, such as seeing through surfaces with radio waves.

As robots and autonomous systems proliferate, equipping them with these advanced computer vision capabilities will be essential. AI-powered perception will allow machines to navigate through and interact with complex unstructured environments and collaborate more smoothly with humans in domains like transportation, manufacturing, healthcare, and services.

Future Outlook and Implications

Looking ahead, we can expect to see "wireless AI" approaches like RF-Pose scale up and find their way into more devices and applications. Researchers will refine the neural network architectures, expand the training datasets, and shrink the specialized RF hardware. Commercial tech giants like Google, Apple and Facebook are also investing heavily in this space.

With further development, RF and wifi-based AIs could achieve even finer-grained pose and activity recognition through walls over wider areas. The technology could be miniaturized into a small IoT gadget for in-home monitoring, or scaled up to a building-wide system for seamless tracking of people and assets.

On one hand, these smart sensing abilities could bring many societal benefits – making hospitals and homes safer, transforming gaming and entertainment, optimizing energy usage in buildings, and assisting first responders. The COVID-19 pandemic has also highlighted the value of remote patient monitoring and disease surveillance tools.

On the other hand, RF-Pose raises tricky ethical questions around privacy, surveillance, and data rights. How would you feel knowing this kind of technology could be watching you in your own home or a public space, even if you‘re not directly visible? Who should be allowed to operate and access such systems, and how do we enforce responsible use and oversight? There‘s a fine line between helpful monitoring and invasive tracking.

As with all cutting-edge AI advances, society will need proactive governance frameworks to mitigate risks and unintended consequences. Researchers and policymakers must work together to ensure these perception tools are deployed transparently, with adequate safeguards, and for legitimate purposes that benefit humanity.

Conclusion

RF-Pose offers an exciting yet unsettling glimpse of an AI-augmented future, where computers can observe and understand our every move using pervasive, penetrating sensors. It‘s a world where the boundary between human and machine perception is blurring, and notions of privacy and identity are shifting.

The CSAIL team‘s neural network that can "see through walls" symbolizes the power and peril of AI as it evolves from narrow object recognition to superhuman perceptual abilities. By learning to interpret subtle signals beyond the limits of human senses, AI is gaining an ever clearer and broader picture of our bodies, behaviors and lives.

Whether this enhanced perception leads to a dream or nightmare scenario depends on how we as a society choose to harness it. But one thing is clear – as AI opens its eyes wider and peers deeper, we‘ll need more than walls to hide from its gaze. Achieving a balance of innovation and ethics in the age of inescapable AI "vision" is one of the grand challenges of our time.