AI Guardman: Using Pose Estimation and Machine Learning to Combat Shoplifting

Introduction

Shoplifting is a pervasive issue in the retail industry, accounting for billions of dollars in losses each year. According to the National Retail Federation, inventory shrinkage cost U.S. retailers an estimated \$61.7 billion in 2019, with shoplifting making up 36.5% of those losses.[^1] While security cameras are a common deterrent, monitoring feeds manually is labor-intensive and prone to human error.

AI Guardman, developed by Japanese startup Earth Eyes in collaboration with NTT East, aims to solve this problem by using artificial intelligence to automatically detect potential shoplifters in real-time. The system applies advanced pose estimation techniques to security camera footage, analyzing body language and movements to identify suspicious behavior and alert store staff via a mobile app.

In this article, we‘ll delve into the technical details of how AI Guardman works, explore the benefits and challenges of using machine learning for retail loss prevention, and discuss the broader implications for AI in physical security applications. We‘ll also examine the latest developments and future outlook for this technology as of 2024.

How AI Guardman Detects Suspicious Behavior

At its core, AI Guardman relies on pose estimation, a computer vision technique that involves detecting the position and orientation of human body joints in an image or video. Pose estimation has been a key focus of AI research in recent years, with applications ranging from augmented reality to robotics and autonomous vehicles.

One of the most well-known pose estimation frameworks is OpenPose[^2], developed by researchers at Carnegie Mellon University in 2017. OpenPose uses a multi-stage deep learning pipeline consisting of a VGGNet-based feature extractor followed by several stages of convolutional layers to predict the 2D locations of up to 135 keypoints on the body, including 25 main joints and additional points for the hands, feet, and face.

AI Guardman takes a similar approach, leveraging convolutional neural networks (CNNs) and large annotated datasets like COCO[^3] and MPII Human Pose[^4] to train its pose estimation model. The system extracts a simplified skeleton consisting of 18 key joints for each detected person in a given video frame.

From there, AI Guardman analyzes the skeletal data to identify predefined poses and movement patterns associated with potential shoplifting behavior. These may include:

• Repeatedly picking up and putting back items
• Looking around furtively or appearing nervous
• Concealing objects in pockets, bags, or under clothing
• Walking quickly or erratically, especially when exiting the store

When the system detects a combination of these behaviors that crosses a certain risk threshold, it sends an alert to the linked smartphone app, providing a video clip and screenshot for staff to review. The app includes a store layout map that highlights areas where suspicious activity has been detected.

Benefits of AI-Based Shoplifting Detection

Early trials of AI Guardman in 2018 showed a 40% reduction in shoplifting incidents at test locations.[^5] While updated figures have not been disclosed, the technology has since been adopted by over 300 retailers across Japan, including convenience stores, supermarkets, and department stores.

The primary advantage of AI-based shoplifting detection is the ability to continuously monitor an entire store without tying up human resources. With traditional CCTV systems, loss prevention staff can only watch one or two camera feeds at a time, making it easy to miss incidents. AI Guardman acts as a tireless, always-on watchdog, flagging potential shoplifters that human observers might overlook.

By focusing on body language and movement rather than facial recognition, AI Guardman also sidesteps some of the privacy and bias concerns associated with identifying individuals. The system does not retain any personal data or match detected individuals against criminal databases.

Retailers who have deployed AI Guardman report positive results beyond just reducing shrinkage. Some have seen increased customer satisfaction and spend, likely due to a greater sense of safety in stores with the technology. Staff also report feeling more secure and supported in their jobs.

Challenges and Ethical Considerations

Despite the promising early results, implementing AI-based shoplifting detection systems raises a number of challenges and ethical concerns that must be carefully addressed.

One key issue is the risk of false positives and unintended bias. While shoplifters can be of any age, gender, or race, AI models may be more likely to flag certain demographics as high-risk due to biases in the training data or assumptions built into the algorithms. This could lead to innocent customers being wrongly identified as potential criminals based on their appearance or mannerisms.

To mitigate this risk, the AI Guardman developers have taken steps to ensure their training datasets are diverse and representative. They also built in safeguards to prevent the system from making automated accusations or taking any punitive actions. The final decision to approach a flagged individual always rests with human staff, who are trained to assess the full context of the situation.

However, even with these precautions, being falsely flagged as a potential shoplifter could be a distressing experience that erodes customer trust. Retailers will need to have clear and transparent policies in place for handling AI-generated alerts and provide a straightforward process for customers to report any issues or complaints.

There are also valid concerns around data privacy and security. While AI Guardman does not collect personal information, the video footage it analyzes could still be considered sensitive data. Retailers must ensure they have robust safeguards in place to protect against unauthorized access or misuse of this data, such as encryption, access controls, and strict retention policies.

As with any AI system deployed in the real world, there is also the risk of the technology being hacked, manipulated, or abused. Criminals could potentially find ways to exploit the system‘s detection rules to evade alerts or even trigger false alarms as a distraction. Regular security audits and penetration testing will be essential to identify and patch any vulnerabilities.

Latest Developments and Future Outlook

Since its initial rollout, Earth Eyes has continued to refine and enhance AI Guardman‘s capabilities. In 2021, the company introduced a new version that incorporates multi-camera tracking to better handle cases where suspects move between different areas of the store.[^6] The updated system can also estimate the value of items potentially being stolen by integrating with store inventory systems.

Other planned enhancements include using facial recognition to detect known shoplifters and analyzing customer behavior patterns to optimize store layouts and product placement. However, these features are likely to face additional scrutiny around privacy and ethics.

Looking beyond shoplifting prevention, the underlying pose estimation technology behind AI Guardman has potential applications in other areas of retail, such as:

• Analyzing customer movement patterns and engagement with product displays to inform merchandising decisions
• Assessing queue lengths and wait times to optimize staffing and checkout placement
• Monitoring for slip and fall hazards or other safety issues in real-time

As of 2024, Earth Eyes has begun pilot testing AI Guardman in other commercial settings, such as office buildings and public transportation hubs. In one trial with the Tokyo Metro, the system was used to detect pickpocketing at busy train stations, with promising early results.[^7]

Other startups are also exploring similar computer vision-based approaches to physical security. U.S.-based Actuate AI has developed a system that analyzes CCTV footage for a range of suspicious behaviors, from shoplifting to vandalism and assault.[^8] China‘s SenseTime offers a "Smart Sentry" product that uses facial recognition and behavior analysis to detect potential threats in public spaces.[^9]

As the technology continues to advance, we can expect to see growing adoption of AI-powered security systems across a range of industries. However, this will also likely lead to increased scrutiny from regulators and the public around privacy, fairness, and accountability.

In 2021, the European Union proposed new regulations that would ban the use of AI for real-time facial recognition in public spaces and impose strict rules on "high-risk" applications like crime prediction.[^10] Other countries are likely to follow suit with their own AI governance frameworks in the coming years.

Conclusion

AI Guardman offers a compelling case study of how machine learning can be applied to real-world security challenges in the retail sector and beyond. By leveraging pose estimation to identify suspicious behavior, the system has the potential to significantly reduce shoplifting losses and improve safety for customers and staff alike.

However, as with any AI technology deployed in sensitive contexts, it is critical that the benefits are carefully weighed against the risks and potential unintended consequences. Retailers looking to implement AI-based loss prevention systems must prioritize transparency, fairness, and accountability at every stage, from system design to staff training and customer communication.

Policymakers and industry leaders will also need to work together to develop appropriate governance frameworks and best practices to ensure this powerful technology is used responsibly as it continues to evolve and spread to new domains. Only by proactively addressing the ethical and societal implications can we harness the full potential of AI to build safer, more secure communities for all.

References

[^1]: National Retail Federation. (2020). 2020 National Retail Security Survey. https://nrf.com/research/2020-national-retail-security-survey
[^2]: Cao, Z., Hidalgo, G., Simon, T., Wei, S. E., & Sheikh, Y. (2018). OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. IEEE transactions on pattern analysis and machine intelligence, 43(1), 172-186.
[^3]: Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., … & Zitnick, C. L. (2014, September). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740-755). Springer, Cham.
[^4]: Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014, September). 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (pp. 3686-3693).
[^5]: IT Media. (2018). AI Guardman shoplifting prevention system reduced theft by 40% in trials. (Japanese) https://www.itmedia.co.jp/news/articles/1805/28/news085.html
[^6]: Earth Eyes. (2021). Earth Eyes releases AI Guardman 2.0 with multi-camera tracking and item detection. (Japanese) https://www.eartheyes.co.jp/news/20210115/
[^7]: Nikkei. (2023). Tokyo Metro tests AI system to catch pickpockets. (Japanese) https://www.nikkei.com/article/DGXZQOUC135340T10C23A1000000/
[^8]: Actuate AI. (n.d.). Actuate AI – Computer Vision for Safer Spaces. Retrieved April 28, 2023, from https://actuate.ai/
[^9]: SenseTime. (n.d.). SenseTime Smart Sentry. Retrieved April 28, 2023, from https://www.sensetime.com/en/product-detail/18
[^10]: European Commission. (2021). Proposal for a Regulation laying down harmonised rules on artificial intelligence. https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence