Machine Learning Draws the Future of Digital Art

The world of art and technology are colliding in exciting new ways thanks to recent advances in machine learning (ML) and artificial intelligence (AI). In particular, ML is now enabling applications and tools that can assist people with drawing and even predict and auto-complete sketches. This technology is putting the power of professional-level drawing into the hands of amateurs and opening up new creative possibilities for artists.

At its core, ML is all about training computer algorithms to learn patterns from data and make predictions. In the case of drawing, ML models can be trained on large datasets of sketches and drawings to learn the common characteristics and structure of different objects and scenes. The model can then analyze a user‘s rough sketch, predict what they are trying to draw, and provide guidance or automatically complete the drawing.

How Machine Learning Recognizes and Generates Drawings

So how exactly does a neural network learn to recognize and generate drawings? The key is a type of model called a convolutional neural network (CNN). CNNs are particularly well-suited for image recognition tasks because they are able to learn hierarchical features. In the early layers, the CNN learns to recognize simple elements like lines, curves and shapes. The deeper layers can then combine these elements to identify more complex structures and patterns.

Training a CNN for drawing recognition typically involves feeding it a large dataset of labeled sketches (e.g. "cat", "house", "tree"). The model compares its predictions to the correct labels and gradually adjusts its internal parameters to minimize the error. With enough training data, the CNN can learn to accurately classify new sketches it hasn‘t seen before.

Generative models like variational autoencoders (VAEs) and generative adversarial networks (GANs) take this a step further. Instead of just recognizing drawings, they can generate entirely new sketches from scratch or based on user input. VAEs learn to encode drawings into a compressed latent space and then decode them back into images. GANs use two competing networks – a generator that tries to produce realistic drawings and a discriminator that tries to spot the fakes. Through many iterations, the generator learns to create images that can fool the discriminator.

Examples of ML-Assisted Drawing Tools

One of the most impressive examples of ML being used for drawing is the SketchAR application. With SketchAR, a user simply has to draw a very basic outline or sketch of an object. The app then uses computer vision to analyze the sketch and predict what the user is trying to draw. It provides an outline that the user can trace over to complete the drawing with more realistic details and proportions.

Under the hood, SketchAR is powered by sophisticated CNNs and computer vision algorithms. The key innovation is a technique called "progressive markers" which allows the app to segment the different elements in its camera view (the drawing surface, the user‘s hand, the drawing instrument, etc.). By isolating just the drawing, the algorithm can map the user‘s sketch to a reference image and guide them to complete the drawing accurately.

Another example is AutoDraw, an experimental web-based tool from Google. With AutoDraw, the user starts sketching an object or scene and the algorithm tries to predict what they are drawing in real-time. It then provides clipart-style suggestions that the user can select to replace their sketch with a more polished version. Behind the scenes, AutoDraw uses a CNN called PoseNet to estimate the object pose and k-nearest neighbors to find the closest matches in its database of professional drawings.

Other recent examples of ML-assisted drawing include:

• Pix2Pix: An app that can automatically convert simple line drawings to photorealistic images using conditional GANs.
• Edges2Cats: A fun web experiment that uses a Pix2Pix model to transform doodles into cat pictures.
• Sketch-RNN: A recurrent neural network (RNN) developed by Google that can complete partial sketches, add creative details, and generate new drawings of common objects.

According to a 2020 paper from MIT researchers, state-of-the-art GANs can now generate drawings that humans have a hard time distinguishing from hand-drawn sketches. In an experiment, they found that people correctly identified the AI-generated drawings only 52% of the time – not much better than random guessing.

Impact on Artists and the Creative Economy

As ML-assisted drawing tools become more accessible and powerful, they are likely to have a profound impact on the creative industries. On one hand, they could democratize art by empowering amateurs to create more impressive works with less time and training. A 2019 survey by the Artists Rights Society found that 75% of respondents were interested in using AI tools to assist their creative process.

However, some professional artists worry that the technology could devalue their skills and disrupt the traditional art market. If anyone can generate near-photorealistic images from simple sketches, will people still pay for original, handmade works? There are also concerns about AI perpetuating biases and aesthetic preferences reflected in the training data, potentially leading to a homogenization of styles.

Adapting to an AI-augmented art world will require new business models, licensing frameworks, and approaches to education. Some experts predict a bifurcation between "slow" and "fast" art – with the former emphasizing the unique, personal touch of human artists and the latter prioritizing rapid experimentation and iteration with AI tools.

"I don‘t see AI as a threat, but rather as an opportunity," says Mario Klingemann, an artist who has been collaborating with GANs and other neural networks for several years. "It allows me to be more playful and take more risks in my work. I can explore ideas that would be too time-consuming or technically challenging to execute by hand."

Other artists are proactively defining the terms of human-AI collaboration and staking out new creative roles. In 2018, Sougwen Chung launched a project called Drawing Operations, in which she "trains" an industrial robot arm to sketch by demonstration – almost like a human apprentice. By programming her own drawing style into the machine, she maintains authorship while also exploring novel partnerings of organic and synthetic creativity.

Beyond Drawing: The Future of AI-Assisted Creativity

Looking beyond drawing and illustration, ML techniques are also being applied to other creative domains such as music, writing, and 3D modeling. Some recent examples:

• Magenta Studio: A suite of ML tools from Google that can help musicians generate melodies, harmonize with a backing track, or morph between different musical styles.
• GPT-3: A massive language model from OpenAI that can automatically generate human-like text on almost any topic with just a small amount of input or context. It has been used to write poetry, dialogue, and even computer code.
• GauGAN: A generative model from NVIDIA that can convert simple doodles and shapes into photorealistic landscape images, complete with textures and lighting effects.

As these AI-assisted tools become more sophisticated and widespread, they could fundamentally change the way we think about creativity and artistic expression. Rather than replacing human artists, they are more likely to augment and accelerate the creative process – helping us explore new aesthetic possibilities, iterate faster, and push the boundaries of what‘s possible.

Some experts even predict the emergence of entirely new art forms and genres enabled by human-AI collaboration. For example, artist Scott Eaton envisions a near future where game engines and virtual worlds are used as "found environments" for AI agents to learn from and evolve in. The resulting behaviors and artifacts could be curated and elevated to the status of art.

Of course, the rise of AI-generated content also raises important ethical questions and challenges. How do we properly attribute authorship and give credit to the human artists involved in training the models? How do we detect and mitigate biases or offensive outputs? And what happens when AI systems become capable of mimicking the style of living artists without their consent?

Responsible development and deployment of these technologies will require ongoing dialogue between artists, technologists, legal experts, and the general public. We need to proactively shape the future we want to see – one in which AI empowers human creativity rather than diminishing it.

Conclusion

Machine learning is already beginning to transform the way we create and interact with art, and the possibilities are only just beginning to be explored. Tools like SketchAR and AutoDraw offer a tantalizing glimpse of a future in which advanced AI systems can serve as creative partners for artists of all skill levels.

While there are valid concerns and uncertainties about the impact of this technology, I believe the overall trajectory is positive. By augmenting and accelerating human creativity, AI has the potential to make art more accessible, expressive, and reflective of our shared values and aspirations.

As an artist and technologist myself, I am excited to be part of this grand experiment at the intersection of human and machine intelligence. I can‘t wait to see what new masterpieces will be drawn – by bits and brushstrokes alike – in the years ahead.