Turn Your Machine Learning Code into a Cartoon Avatar Generator with Mercury

Jupyter notebooks have become an essential tool for data scientists and machine learning engineers. They provide an interactive environment to write and execute code, visualize data, and share results. But did you know you can also turn your Jupyter notebook into a web app that anyone can use?

In this post, we‘ll walk through how to build a "toonify" app that turns photos into cartoon avatars, all within a Jupyter notebook. We‘ll then convert this notebook into an app in just a few steps using a tool called Mercury. By the end, you‘ll see how you can make your machine learning code interactive and accessible to a wider audience.

What is Toonification?

Toonification is the process of transforming an image to make it look more like a cartoon or comic. This effect has become popular in social media and messaging apps that generate user avatars.

Under the hood, toonification is typically done using machine learning techniques like generative adversarial networks (GANs). A GAN consists of two neural networks – a generator that creates new images and a discriminator that tries to distinguish real images from generated ones. By training these networks on cartoon images, the generator learns to produce realistic cartoon versions of input photos.

One popular toonification model is AnimeGAN, which was developed by researchers at Huawei in 2020. AnimeGAN uses a specialized GAN architecture and training process to generate high-quality cartoon images with smooth lines and shading. We‘ll be using a PyTorch implementation of AnimeGAN called AnimeGAN2 for our app.

Building the Toonify App in a Jupyter Notebook

Let‘s dive into building the toonify app step-by-step within a Jupyter notebook.

Step 1: Install Dependencies

First, create a new Jupyter notebook and install the required libraries by running the following commands in a code cell:

!pip install mljar-mercury
!pip install torch torchvision
!pip install Pillow

This will install Mercury for converting the notebook to an app, PyTorch for running the AnimeGAN model, and Pillow for image processing.

Step 2: Import Libraries

In the next code cell, import the installed libraries:

import os
import torch
from PIL import Image

Step 3: Load Pre-trained Model

Download the pre-trained AnimeGAN2 model from GitHub and load it using PyTorch:

model = torch.hub.load("bryandlee/animegan2-pytorch:main", "generator", pretrained="face_paint_512_v1")
face2paint = torch.hub.load("bryandlee/animegan2-pytorch:main", "face2paint", size=512)

This code loads the generator component of AnimeGAN2 which will transform the input images. The face2paint function handles resizing and preprocessing the images before passing them to the generator.

Step 4: Define Toonify Function

Next, let‘s define the toonify function that will take an input image and return the toonified version:

def toonify(img, model, face2paint):
out = face2paint(model, img)
return out

This function simply passes the input image through face2paint and the generator model and returns the output cartoon image.

Step 5: Set Up Mercury App

Now we‘re ready to turn our notebook into an app using Mercury. At the top of the notebook, add the following YAML configuration in a raw cell:

---

title: My Toonify App
description: Turn photos into cartoon avatars
params:
image_file:
input: file
label: Select an image
type: image
size: 512×512
output: image

This configuration specifies the app title, description, input parameter (the user will upload an image file), and output type (an image). Mercury will generate a user interface with an image upload button based on this.

Step 6: Process User Input

Finally, we‘ll add a code cell to load the user‘s uploaded image, pass it through our toonify function, and display the output:

if ‘image_file‘ in locals():
img = Image.open(image_file)
cartoon_img = toonify(img, model, face2paint)
else:
print(‘Please upload an image!‘)

The if statement checks if the user has uploaded an image. If so, it loads the image, runs it through toonify to generate a cartoon version, and displays it. The output will appear in Mercury‘s generated UI.

Converting the Notebook to an App with Mercury

With the notebook complete, we‘re ready to convert it to an app using Mercury. Simply open a terminal, navigate to the directory containing the notebook file, and run:

mercury run notebook.ipynb

Replace notebook.ipynb with the actual name of your notebook file.

This will launch Mercury which will generate a web UI for your app. By default, it will be hosted locally at 127.0.0.1:8000. Open this URL in a web browser to use your toonify app!

Mercury scans your notebook for the YAML configuration and input/output cells and builds the UI around them. It allows users to upload an image, runs the code to toonify it, and displays the generated cartoon image, all through an intuitive interface.

Deploying the App on Hugging Face Spaces

To make your toonify app accessible to anyone on the web, you can deploy it for free on Hugging Face Spaces. Here‘s how:

1. Create an account on huggingface.co if you don‘t already have one.

2. Click "Spaces" in the top navigation and then "Create new Space".

3. Fill out the form to create a space for your app. Choose "Gradio" for the SDK.

4. In your new space, upload your notebook file, a requirements.txt listing your dependencies, and an app.py file to load the notebook as a Mercury app.

5. Add a model card and README describing your app and hit "Build".

Hugging Face will automatically build and deploy your app, making it accessible on the web at a huggingface.co/spaces URL. Share this link so anyone can toonify their photos!

Potential Applications and Benefits

Toonification is a fun application of machine learning that can be used for more than just generating social media avatars. Some potential applications include:

• Education: Toonifying historical figures or book characters could make learning more engaging for students
• Entertainment: Cartoon versions of movie scenes or celebrity photos could be used for parody or fan art
• Digital art: Artists could use toonification to inspire new character designs or styles
• Accessibility: Toonifying icons and interfaces could make them easier to recognize for those with visual impairments

From a technical perspective, converting Jupyter notebooks to apps with Mercury offers several benefits:

• Shareability: Packaged apps are easier to distribute and use compared to raw code
• Interactivity: Apps allow users to experiment with different inputs and see results in real-time
• Abstraction: App users don‘t need to understand the underlying code, making it accessible to non-technical audiences
• Reproducibility: Notebooks and Mercury configs document the exact code and environment used, making apps easy to reproduce

Limitations and Areas for Improvement

While powerful, toonification and Mercury apps do have some limitations to consider:

Toonification quality depends heavily on the training data and model architecture. Anime GAN provides good results but other models trained on different cartoon styles could improve quality and variety.

Mercury currently only supports a limited set of input types and layouts. More customization options for the UI could make apps more engaging.

Free Hugging Spaces and Gradio deployments may have usage limits for heavier apps. Paid tiers or other platforms may be needed for high traffic.

To take your toonify app further, you could experiment with different GAN models, add options to control the cartoon style, or even allow users to train their own models. You could also enhance the UI with Mercury‘s more advanced features or connect your app to other APIs and services.

Conclusion

We‘ve seen how to build a toonify app that turns photos into cartoons, all within a Jupyter notebook using AnimeGAN. By converting this notebook into an app with Mercury and deploying it on Hugging Face Spaces, you can make your machine learning model interactive and accessible to anyone.

Toonification is just one example of the creative possibilities unlocked by deep learning and the power of Jupyter notebooks as an interface for machine learning. As tools like Mercury make it easier to productionize notebooks, we can expect to see many more fascinating AI-powered apps in the future.

So why not give it a try? Grab a Jupyter notebook, pick a fun machine learning model, and see how you can turn your research code into an engaging app for the world to play with. The possibilities are endless!