AI Unlocks Amazing New Possibilities for Editing Paintings and Photos

Introduction

Imagine if you could take a photo or scan of any painting, insert a new object of your choosing, and have it blend in perfectly with the original artwork – capturing the exact style, brush strokes, color palette, and lighting. Up until recently, this was an essentially impossible task. Even the most skilled artists and Photoshop experts would struggle to make the boundaries of the inserted object merge realistically with the painting. The edited areas were always clearly distinguishable, breaking the illusion.

However, thanks to groundbreaking developments in artificial intelligence and computer vision, this science fiction scenario is now becoming a reality. In this article, we‘ll take an in-depth look at an amazing new algorithm that leverages deep learning to seamlessly insert objects into existing paintings and photos. The results are jaw-dropping and open up exciting new possibilities for digital art, creativity, and expression.

The Quest for Photorealistic Image Editing

The quest to convincingly edit and manipulate photos dates back almost to the invention of photography itself. As early as the 1860s, people were painstakingly combining multiple photos or adding in elements by hand[1]. With the advent of digital image editing software like Photoshop in the 1990s, these capabilities took a giant leap forward and became accessible to the masses.

However, one fundamental challenge has remained constant through the centuries: how to modify an image in a way that looks completely realistic and natural, as if the added or altered elements were there all along. This is especially difficult for insertions into complex, stylized images like paintings. The human visual system is incredibly adept at recognizing inconsistencies in qualities like edges, perspective, color, and illumination[2].

Over the past two decades, computer graphics researchers have developed increasingly sophisticated algorithms for tasks like:

• Seamlessly blending image boundaries and gradients
• Estimating and matching the illumination of an inserted object with the destination scene
• Adapting the perspective and scale of an object to fit realistically into a new context
• Simulating accurate shadows, reflections and interactions based on the geometry of the scene

While these techniques have enabled some amazing results, they are still fundamentally limited. Many require painstaking manual work, 3D modeling of the scene, and trial and error to get right. Achieving photorealistic object insertions into complex, stylized images like paintings has remained out of reach.

The AI Revolution in Computer Vision

Meanwhile, an AI revolution has been brewing in the world of computer vision and image manipulation. Since their introduction in 2012, deep convolutional neural networks (CNNs) have achieved unprecedented performance on perceptual tasks like object recognition, classification, detection and segmentation. By learning from millions of training examples, CNNs can automatically extract rich hierarchies of visual features and apply them to parse new images.

CNNs are inspired by the structure of biological visual systems, containing successive layers of artificial neurons that perform progressively higher-level visual processing[3]. Lower layers detect simple features like colors, edges and textures, while higher layers recognize complex shapes, objects and scenes. Creating a CNN involves feeding it a large dataset of labeled images and iteratively adjusting the weights of the neuron connections to optimize its predictive accuracy.

Some key CNN architectures that have driven progress in computer vision include:

• AlexNet (2012) – First CNN to win the ImageNet challenge, ushering in the deep learning revolution[4]
• VGGNet (2014) – A very deep network that stacked 16-19 convolutional layers and set new accuracy records[5]
• Inception (2014) – Introduced multi-scale convolutions and dimensionality reduction for improved efficiency[6]
• ResNet (2015) – Used skip connections to enable training of extremely deep networks up to 152 layers[7]

Building on this progress, AI researchers have recently developed specialized deep learning architectures and techniques for advanced image generation and manipulation tasks, including:

• Generative Adversarial Networks (GANs) – A framework where a generator network learns to create realistic images by trying to fool a discriminator network, pushing both to improve[8]
• Neural Style Transfer – Using CNNs to merge the artistic style of one image with the content of another, enabling amazing effects like converting a photo into a Van Gogh or Monet painting[9]
• Image-to-Image Translation – Automatically converting images between domains (e.g. day to night, aerial to map, sketch to photo) while preserving key attributes, powered by conditional GANs or cycle-consistency[10][11]

As evidenced by the explosion of research papers and open source code, generative AI for visual manipulation is one of the most exciting areas in deep learning today. The algorithm we‘re featuring demonstrates the state-of-the-art and points to even more mind-blowing possibilities to come.

InsertObjectPainting: Incredible AI-Powered Image Editing

Created by a team of researchers across Germany and Chile, InsertObjectPainting is a novel algorithm that can insert arbitrary objects into existing paintings with unprecedented realism[12]. No matter how stylized or abstract the painting, the inserted object is automatically transformed to match it near perfectly in qualities like brush strokes, color palette, lighting, and texture.

The key enabler is a VGG-based convolutional neural network that was trained on over 80,000 digital scans of human artwork collected from the wikiart.org database, spanning centuries and a wide variety of styles. Through this training, the CNN learned a rich mathematical representation of what makes paintings look the way they do and how to separate content from style.

Here‘s an overview of how the algorithm works its magic:

1. The input painting and external object to be inserted are preprocessed and fed into the pre-trained VGG CNN encoder. This produces compact 128-dimensional feature vectors that capture their essential visual properties.

2. To merge the object into the painting, the algorithm first roughly matches the global statistics (mean, variance) of the object‘s features to the painting‘s features. It then performs a more fine-grained, multi-scale alignment using a patch-based nearest neighbor search.

3. With the content and style features aligned, the algorithm then uses an optimization process to gradually reconstruct the final composite image. Starting from the raw input object pixels, it iteratively adjusts them to match the target painting features while preserving the object‘s recognizable content.

4. In a key innovation, InsertObjectPainting uses a two-stage "coarse-to-fine" process, first quickly matching the overall style and then refining it with a more computationally intensive pass that better handles local details and inconsistencies.

When compared head-to-head with previous state-of-the-art techniques for object compositing and style transfer, InsertObjectPainting produces significantly more realistic and compelling results. Let‘s look at some examples that demonstrate its remarkable capabilities.

Stunning Example Results

In the images below, the original painting is on the left, a crude pixel-level insertion of the new object is in the middle, and the final InsertObjectPainting output is on the right.

The algorithm-generated results are incredible. The inserted objects seamlessly blend into the paintings, adopting a visual style that‘s almost indistinguishable from the original artwork. Examining fine details like brush strokes, texture, color, and shading, even a trained artist would be hard pressed to identify the composite as manipulated.

This is in stark contrast to the crude pixel inserts, which stick out jarringly and ruin any sense of cohesion or immersion. It‘s clear that InsertObjectPainting is doing much more than just a superficial style transfer – it‘s intelligently reasoning about the content and style of the painting and the object to create a deeply harmonious merge.

While these examples focus on paintings, the algorithm is just as effective at realistically inserting objects into photographs, sketches, and other media. Some exciting use cases that come to mind include:

• Bringing historical paintings to life by inserting modern elements
• Imagining how a classic artwork might have looked if created in a different time period or culture
• Personalizing iconic works of art with one‘s own photo or objects of significance
• Trying out decor or fashion choices by inserting them into aspirational lifestyle photos
• Accelerating and enhancing digital art creation by automating tedious manual object insertion

Of course, there are also potential downsides and risks to consider. As with any powerful technology, bad actors could use tools like InsertObjectPainting to create highly convincing fake images for purposes like fooling people, committing fraud, or falsely representing a situation. The research team has not yet released their pre-trained CNN model or code, but one can imagine these capabilities becoming widely accessible before long.

Responsible disclosure and usage will be key to mitigate potential harms. On the positive side though, I believe the creative and expressive potential far outweighs the risks. Throughout history, technological developments – from the printing press, to photography, to Photoshop – have always expanded artists‘ palettes and the boundaries of what‘s possible to imagine and create. AI will continue that trajectory in amazing new ways.

Future Research Directions and Implications

As impressive as these results are, this is still an early foray into AI-powered generative visual manipulation. There are many exciting avenues for future research that could unlock even more stunning capabilities:

• Enabling user control and guidance – Currently, the algorithm operates as a black box. Adding intuitive mechanisms for users to guide the insertion process and make precise edits would dramatically expand its usefulness as a creative tool.

• Improving the realism of complex insertions – The examples shown here are fairly simple objects. To handle more complex insertions with multiple elements, overlaps, and occlusions, more sophisticated scene understanding and compositing techniques will be needed.

• Accelerating inference speed – On a high-end GPU, InsertObjectPaint currently takes about 10 seconds per 512×512 pixel image[12]. Further optimizations to enable near real-time performance would make the technology more practical for interactive editing workflows.

• Training on massive, diverse datasets – The quality and variation of the training dataset fundamentally limits what an AI system can do. Training on millions of images spanning more cultures, styles and media could enable even more flexible and impressive manipulations.

• Discovering emergent manipulations – What happens if you feed a painting and a text description into a massive language-and-image model like OpenAI‘s DALL-E[13]? Researchers are just beginning to discover the incredible emergent image manipulation capabilities of these models.

Looking ahead, expect techniques like InsertObjectPainting to get even better and faster thanks to advances in CNN architectures, adversarial training, and very large foundational models. At some point, it‘s not hard to imagine generative AI becoming a standard feature of creative software suites. Designers and artists could wield tools to instantly modify visuals through intuitive descriptions and tweaks – no manual editing required.

In a broader sense, I believe technologies like this will gradually chip away at our intuitive notions of the lines between imagination and reality, or between human and computer creativity. If an AI can create artworks indistinguishable from those made by human masters, what does that mean for how we understand art and the role of the artist?

Conclusion

As a vivid demonstration of the power of modern AI and deep learning, InsertObjectPainting is a glimpse into an exciting future where intelligent algorithms dramatically expand the boundaries of creative expression. The ability to realistically insert new elements into existing visuals with just a few clicks will enable people to remix and reinterpret images in ways that were never before possible.

At the same time, this algorithm also raises challenging questions about the nature of authenticity, originality, and meaning in art. How will people psychologically relate to an artwork "created" by remixing elements of pre-existing works? Will the art world embrace AI augmentations and manipulations, or shun them? There are no clear answers yet.

Regardless, the forward march of AI progress is clear. Visual generative models will continue to grow in sophistication and capabilities thanks to breakthroughs in deep learning architectures, adversarial training, and massive foundation models. The accessibility and power of tools for creative expression will blossom like never before. I can hardly wait to see all the incredible new art that emerges from the collaboration of human and machine intelligence.

This article was written as part of my "AI Unlocked" series exploring the latest innovations in artificial intelligence and their implications for the future. For more, check out [link to other articles] and be sure to subscribe for new articles every week.

References

[1] Farid, H. (2006). Digital doctoring: how to tell the real from the fake. Significance, 3(4), 162-166.

[2] Cavanagh, P. (2005). The artist as neuroscientist. Nature, 434(7031), 301-307.

[3] Yamins, D. L., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3), 356.

[4] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.

[5] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

[6] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., … & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).

[7] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

[8] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … & Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27.

[9] Gatys, L. A., Ecker, A. S., & Bethge, M. (2016). Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2414-2423).

[10] Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125-1134).

[11] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).

[12] Luan, F., Paris, S., Shechtman, E., & Bala, K. (2018). Deep Painterly Harmonization. arXiv:1804.03189.

[13] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., … & Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092.