Meta Open Sources AI Model That Brings Doodles to Life

In a significant step forward for creative AI, Meta (formerly Facebook) has open sourced its Animated Drawings model – a cutting-edge machine learning system that can automatically bring children‘s sketches and doodles to life. By analyzing a static drawing and identifying humanoid characters, the AI can construct a movable puppet that dances, jumps, and moves in lifelike ways.

Meta first previewed this technology in 2021 with a prototype web application that let users upload their drawings and watch them get animated in real-time. During the initial demo phase, the system processed over 6.7 million user-submitted sketches, which Meta used to further train and refine the model. Now, the company has taken the momentous step of releasing the model‘s code and training data to the public, inviting researchers and developers worldwide to experiment with the system and contribute improvements.

How It Works

At its core, Meta‘s Animated Drawings model leverages state-of-the-art techniques in computer vision, human pose estimation, and motion synthesis to turn static drawings into animated characters. The process involves several key steps:

1. Object Detection: The model uses a convolutional neural network (CNN) to analyze the input image and locate any humanoid figures. This is challenging with children‘s drawings, which often contain abstract shapes and scribbles that don‘t resemble realistic anatomy. Meta‘s researchers developed a novel architecture called HumanFinder that is specifically trained to detect childlike drawings of people.

2. Part Segmentation: Once a human figure is detected, the model needs to segment it into different body parts like the head, torso, arms, and legs. This is done using a graph convolutional network (GCN) that analyzes not just the visual appearance of each part, but also its spatial relationships and connectivity to other parts. The GCN is trained on a large dataset of manually annotated character drawings to learn the common "building blocks" and patterns found in human sketches.

3. Pose Estimation: With the character segmented into parts, the model then estimates its 2D joint positions and skeletal pose. This involves predicting the locations of key joints like the neck, shoulders, elbows, wrists, hips, knees, and ankles. Meta‘s model uses a hierarchical pose estimation approach that first predicts a coarse global pose and then refines it with local joint predictions.

4. Geometry Extraction: To create a movable puppet, the model needs to extract clean vector geometry from the messy raster sketch. This is done using a combination of techniques, including contour detection, polygon simplification, and curve fitting. The goal is to create a set of connected 2D shapes that closely match the original drawing but with simpler, smoother edges that are easier to manipulate.

5. Texture Transfer: The extracted puppet geometry then gets "skinned" with colors and textures sampled from the original drawing. Meta‘s model uses a style transfer approach based on generative adversarial networks (GANs) to apply the original stroke styles and shading in a coherent way, even if the puppet geometry differs significantly from the sketch.

6. Motion Synthesis: Finally, the textured puppet is brought to life with animation. Meta‘s model supports a variety of motion synthesis techniques, including keyframe animation, physics-based simulation, and deep learning methods like motion VAEs and autoregressive models. These can be used to generate dances, walks, jumps, and other motions based on predefined styles or controlled by user inputs.

The entire end-to-end pipeline is implemented as a deep neural network that can be trained on large datasets of character drawings and animations. Meta‘s researchers have continued to improve the model‘s accuracy and robustness since the initial demo, achieving state-of-the-art performance on several benchmark tasks. For example, the latest version of HumanFinder achieves a mean average precision (mAP) of 92.7% on detecting humanoid figures in children‘s drawings, a 6.8% improvement over the original model.

Empowering Creative Expression

A key motivation behind Meta‘s Animated Drawings project is to empower people, especially children, to express their creativity in new ways. By open sourcing the model and inviting contributions from the developer community, Meta hopes to spur innovations that make the technology accessible to a wide range of users and use cases.

One promising application is in education, where animated drawings could make lessons more engaging and interactive. Imagine a science textbook where you could sketch a simple machine and watch it come to life to demonstrate physics concepts. Or a history lesson where students could animate their own scenes from a famous battle or event. The ability to visualize and manipulate abstract ideas through drawing could be a powerful learning aid.

Animated Drawings could also unlock new forms of creative expression and storytelling. Instead of just writing a story, children could sketch out their characters and scenes and then animate them to create a fully realized cartoon. Artists and designers could use the technology to rapidly prototype and iterate on character designs and motions. And anyone could turn their doodles into playful animations to share with friends and family.

Meta‘s researchers see this as part of a broader trend towards "AI-augmented creativity", where machine learning tools empower people to create and express themselves in ways that were previously difficult or impossible. By making the Animated Drawings model open source, they hope to accelerate progress in this exciting field and make the benefits accessible to all.

Open Source Impact

Since releasing the Animated Drawings model and dataset to the public on GitHub, Meta has seen significant engagement from the developer community. The repository has garnered over 5,000 stars and 500 forks, with dozens of pull requests proposing improvements and extensions to the code.

One popular contribution adds support for animating multi-character scenes, allowing users to draw and interact with multiple figures at once. Another enables the model to extract and animate simple facial expressions from the drawings, adding an extra layer of emotional expressiveness. And several developers have experimented with integrating the model into creative apps and games, from storybook creators to virtual pets.

The open source dataset has also proven valuable for research into AI fairness and bias. By analyzing the demographic diversity of the character drawings, researchers can study how well the model performs on different genders, races, ages, and artistic styles. This can help identify potential biases and guide efforts to make the technology more inclusive.

Meta‘s own researchers have continued to refine the model and explore new use cases since the open source release. One recent project trained the model to recognize and animate simple 3D characters constructed from basic shapes like spheres, cubes, and cylinders. This points the way towards a more flexible and expressive animation system that could bring all kinds of objects to life, not just humanoid figures.

The Future of AI-Augmented Creativity

Meta‘s Animated Drawings project is just one example of the incredible progress being made at the intersection of artificial intelligence and human creativity. As machine learning models become more sophisticated at understanding and interacting with visual media, they are enabling new forms of expression and storytelling that blur the lines between the physical and digital worlds.

We are already seeing a proliferation of AI-powered creative tools, from text-to-image generators that can conjure up photorealistic scenes from natural language descriptions, to style transfer algorithms that can transform photos and videos into paintings or cartoons. As these technologies advance and become more widely accessible, they could democratize creative production and enable anyone to bring their ideas to life with just a few clicks or sketches.

At the same time, the rise of AI-augmented creativity raises important questions about the role of human artistry and authorship in an age of intelligent machines. Will AI tools empower human creators to achieve new heights of imagination and skill, or will they replace them entirely? How do we ensure that the benefits of these technologies are distributed fairly and that they reflect the diversity of human experience?

These are complex challenges that will require ongoing collaboration between researchers, developers, artists, and policymakers. But one thing is clear: the future of creativity is inextricably linked to the progress of artificial intelligence. By open sourcing its Animated Drawings model, Meta has taken an important step towards a world where anyone can bring their wildest ideas to life, one sketch at a time. As more developers build on this foundation, the possibilities are endless – and endlessly exciting.