Beyond Imitation: 8 Cutting-Edge Data Science Projects Generating Original Art and Music
Introduction
In recent years, the boundaries between artificial intelligence and creativity have blurred as data science projects venture into artistic domains once considered off-limits to machines. From generating photorealistic portraits to composing professional-quality music, AI systems are now producing original creative works that challenge our most basic assumptions about the nature of art and imagination.
This article will explore eight groundbreaking projects at the forefront of AI-generated art and music. We‘ll dive into the machine learning architectures powering these innovations, examine their creative outputs, and consider the wider implications for artists, data scientists and society at large. While the technologies behind these projects are highly sophisticated, the possibilities they represent should excite anyone with an interest in the future of creativity.
Transforming Sketches into Photorealistic Landscapes with GANs
In 2018, researchers at NVIDIA wowed the world with an AI model capable of transforming simple doodles into photorealistic landscape images.[^1] Building on prior work with generative adversarial networks (GANs), the team trained their model on over 1 million pairs of sketches and landscape photos scraped from Flickr.
The architecture used, known as an image-to-image translation GAN, learns a mapping between the low-level features of the input sketch (e.g. lines, curves, shapes) and the high-level semantic content of the corresponding photo (trees, ponds, mountains, etc.).[^2] At inference time, the generator network uses this mapping to construct a realistic landscape from the semantic building blocks of the input sketch.
Compared to earlier GAN-based sketch-to-photo approaches, NVIDIA‘s model produces higher resolution images (up to 1024×1024) while preserving more recognizable details from the original sketch. In a user study, human raters perceived the generated landscapes as real photos 18% of the time– a significant improvement over the previous state-of-the-art.[^1]
This sketch-based interface represents an intuitive and accessible way for users to interact with powerful generative models. Doodling with GANs may be an entertaining demo, but the potential use cases extend far beyond art and entertainment. As the authors note, this approach could help architects, urban planners and interior designers rapidly visualize and iterate on design ideas.
Neural Painters: Fusing Art History and Machine Learning
If you‘ve ever dreamed of painting like Van Gogh or Picasso, a neural style transfer model may be your best bet. Popularized by apps like Prisma and DeepArt, these algorithms excel at rendering photos in the signature brushstrokes and palettes of famous artists.
Neural style transfer is not a new technique– the basic approach has been around since 2015. But recent research has pushed the sophistication and applicability of these models to new heights. A 2019 paper from the University of Maryland introduces a style transfer system that can faithfully emulate a variety of artists, from Cézanne to Ukiyo-e masters, in a single trained model.[^3]
What distinguishes this approach is the use of an invertible neural network architecture for the style transformer. Unlike the feedforward networks typically used for style transfer, invertible models learn to reconstruct both the content and style inputs from the stylized output. This allows them to achieve more precise control over stylization while preserving content from the original image.
The Maryland researchers conducted extensive experiments comparing their model to five previous style transfer approaches. In a user study with 244 participants, the invertible network was rated as producing more visually appealing images that better captured the artists‘ styles.[^3] Such AI-powered tools could allow aspiring digital artists to tap into art history for inspiration and even create new works mimicking the masters.
Jukebox and MuseNet: Generating Music in Any Genre or Style
In April 2019, OpenAI stunned the music world with the release of MuseNet, a deep neural network capable of generating 4-minute musical compositions with 10 different instruments and styles ranging from classical to jazz.[^4] Just one year later, they outdid themselves with Jukebox– an even more ambitious model that generates music, complete with lyrics and vocals, as raw audio in a variety of genres and artist styles.[^5]
Trained on massive datasets of songs and instrument recordings, these models learn hierarchical representations of music that allow them to construct new compositions from the ground up. MuseNet uses an approach called transformers (attention-based neural networks) to capture long-range dependencies in symbolic music data like MIDI files.[^4] Jukebox, on the other hand, works directly with raw audio, using a novel architecture called a Vector Quantized Variational Autoencoder (VQ-VAE) to encode songs while modeling high-level musical qualities like genre, mood and instrumentation.[^5]
The generated outputs from these models are striking in their musical coherence and stylistic fidelity. In a blind listening test, 55% of participants reported that music from Jukebox sounded like it could have been composed by a human artist. Diving into the technical details, it‘s clear that Jukebox‘s multi-level VQ-VAEs are key to its ability to capture musical elements across vastly different timescales, from granular audio textures to overarching song structures.
While these models represent major advances, they have important limitations. The music can sound sonically and thematically disjointed, with abrupt transitions, inconsistent lyrics, and occasional audio glitches.[^5] Addressing these shortcomings will likely require not only bigger models and datasets, but also new machine learning paradigms tailored to the unique challenges of music generation.
Writing Novels and Movie Scripts with Language Models
The same neural architectures behind breakthroughs in machine translation and question answering are now being harnessed for longform creative writing. Large language models like OpenAI‘s GPT-3 can generate impressively fluid and coherent text in a range of literary styles, from poetry to screenplays.
Consider Botnik Studios‘ novel The Seeker, advertised as "the first machine-generated sci-fi book." Released in 2018, this full-length novel was produced by fine-tuning GPT-2 (a predecessor to GPT-3) on a hand-curated selection of classic and contemporary sci-fi literature.[^6] The raw model outputs were then heavily edited and reassembled by human writers into the final narrative.
Other projects have taken a more hands-off approach, using language models as a tool to inspire and augment human creativity rather than fully automate it. Dramatron is an AI co-writing tool that helps playwrights construct scenes, write character dialogues and brainstorm plot ideas.[^7] Developed by a team from the University of Melbourne and RMIT University, Dramatron uses a combination of template filling and large language models to generate script excerpts that writers can adopt, rewrite or simply use as creative fodder.
Similarly, AI Dungeon leverages GPT-3 to power a text adventure game that crafts a unique interactive story based on the player‘s open-ended actions and dialogue.[^8] The scope of the generated narratives is virtually unbounded– players can explore everything from fantasy quests to outer space odysseys to slice-of-life dramas, all while conversing with a cast of AI characters in natural language.
These examples hint at the wider potential of language models to augment human creativity and enable new forms of interactive, generative storytelling. While truly autonomous AI novelists may still be on the horizon, tools for co-creating with language models are poised to become an increasingly common part of many writers‘ workflows.
AI Art Generators: Empowering Digital Artists
The proliferation of user-friendly AI art tools has opened up exciting new possibilities for digital artists and illustrators. With just a few words or sketches as input, models like Midjourney and DALL-E 2 can generate a practically limitless range of images– everything from photorealistic portraits to abstract digital paintings to surrealist fusions of concepts that defy categorization.[^9]
Table 1 compares some of the key features and capabilities of popular AI art generators.
| Model | Input Modalities | Training Data | Max Resolution |
|---|---|---|---|
| DALL-E 2 | Text, Image | 650M image-text pairs from the web | 1024×1024 |
| Stable Diffusion | Text | 2.3B image-text pairs from LAION-5B | 512×512 |
| Midjourney | Text | Private dataset of 1B images | 1664×1664 |
While the technical details of these models are not always fully disclosed, they generally rely on large text-to-image models trained on web-scale datasets (often measured in billions of images).[^9] The open source Stable Diffusion model, for instance, uses a latent diffusion architecture originally developed for image inpainting and super-resolution. This approach iteratively denoises a low-resolution image seed to generate high-fidelity outputs conditioned on a text prompt.[^10]
One key advantage of this newer wave of art generators is their flexibility and controllability. Users can fine-tune the tools on their own images, specify styles and artistic mediums, and even upload a "negative prompt"– a description of visual elements to exclude from the generated image. This allows artists to dial in a desired aesthetic while maintaining a level of creative unpredictability.
As these tools mature and diffuse into the mainstream, they are sparking important conversations around the ethical and legal implications of AI art. Some in the art community have raised concerns about AI systems appropriating the styles of human artists without consent or compensation.[^11] Others worry that AI will devalue or displace the work of professional illustrators, particularly if the tools advance to the point of fully automating commercial art tasks. Developing responsible, equitable frameworks for AI creativity is an urgent challenge that will require input from artists, technologists and policymakers alike.
Conclusions
From doodles to disco hits, the projects surveyed here demonstrate the incredible breadth of what‘s possible when machine learning meets creative expression. But they represent just a few pixels in a much vaster landscape of AI-powered tools and artforms still over the horizon.
In many ways, we are witnessing a new kind of digital renaissance– one in which artists, writers and musicians can harness AI as a collaborator and catalyst for their creative visions. As language models grow more eloquent, as generative algorithms learn to paint with an ever-finer brush, the boundaries between artificial and human imagination will blur further.
These AI creativity tools are not without risks and challenges. We must grapple with complex questions around intellectual property rights, job displacement, and the existential purpose of art in an age of machines that can paint and compose. But if we approach these tools with thoughtfulness and care– as a complement to, rather than replacement for, human creativity– they could open up radically new forms of expression and storytelling.
Ultimately, the story of AI art and music is still being written. What plot twists and poetic turns it will take remains to be seen. But one thing is certain: the future of creativity will not be boring.
