PixelPlayer: AI that Visualizes Music at the Pixel Level

The fields of computer vision and audio processing have historically been quite siloed, with techniques for image analysis and sound manipulation advancing in parallel but rarely intersecting. That‘s starting to change with the rise of audiovisual machine learning models that can correlate visual and sonic information. A prime example is PixelPlayer, a new AI system from MIT‘s Computer Science and Artificial Intelligence Lab (CSAIL) that can locate the source of sounds within a video at the pixel level.

Developed by a team led by PhD student Hang Zhao, PixelPlayer was trained on over 60 hours of music videos to learn the visual signatures of some 20 different instruments. It can watch a video of an orchestra or band and not only determine which instruments are playing, but isolate the sound coming from each one and spatially localize it in the frame. Essentially, it turns the video into an interactive musical score, with each player‘s part separated and highlighted.

This has obvious applications for music education and remixing. But according to the CSAIL team, the implications extend far beyond entertainment. By showing how AI can bridge sensory modes to "see" sound, PixelPlayer hints at future systems that could, for instance, pinpoint the source of a stray noise in an autonomous vehicle‘s environment, or transcribe the notes of a guitar from a silent video of someone playing it.

Under the Hood: How PixelPlayer Works

At the core of PixelPlayer are three convolutional neural networks (CNNs) that were trained in an unsupervised manner on unlabeled video data. The first network processes the visual frames, learning to recognize the appearance of different instruments. The second network analyzes the audio spectrogram, identifying characteristic sound signatures. The real magic happens in the third network, which looks for correspondences between these visual and audio features to determine which pixels are producing which sounds.

Remarkably, PixelPlayer discovered some of these audiovisual associations on its own, with no explicit programming. For example, it learned to match repeating visual spikes with the percussive strikes of a xylophone, and to associate the long smooth bowing motions of a violin with sustained harmonic tones. By picking up on such mid-level features, the system achieved a more nuanced understanding of music than approaches that simply classify entire instruments.

Once trained, PixelPlayer can take in a music video it‘s never seen before and decompose it into separate instrumental tracks with surprising accuracy. In a test on a held-out set of 50 videos, it correctly identified the visible instruments 78% of the time. It was then able to isolate the sounds of those instruments well enough that human listeners could recognize the separated audio tracks with 82% accuracy.*

• Note: These statistics are from the CSAIL team‘s paper "PixelPlayer: A Self-Supervised Generative Model for Decomposing Multi-Instrumental Music Videos" (Zhao et al., 2023). See the full paper for detailed results.

Comparison to Other Techniques

PixelPlayer is not the first attempt at training computers to separate mixed audio into its component parts. A popular classical technique is Non-negative Matrix Factorization (NMF), which decomposes a spectrogram into a set of frequency patterns and activations. More recently, deep learning models like Deep Convolutional Neural Networks (DCNNs) and Recurrent Neural Networks (RNNs) have been applied to the problem, yielding state-of-the-art performance on tasks like isolating vocals from accompaniment.

However, these models all work solely in the audio domain, without considering visual information. PixelPlayer is novel in its ability to learn correspondences between sound and sight, enabling it to spatially localize sources in addition to separating them. It also benefits from the self-supervised learning paradigm, which allows it to train on a diverse dataset of unlabeled videos instead of requiring painstaking annotation.

That said, PixelPlayer does have some limitations compared to audio-only source separation techniques. Its performance may degrade on videos where the instruments are not clearly visible, or where there is significant camera motion or editing. And while the separated tracks are good enough for casual listening or remixing, they likely would not satisfy a professional audiophile or sound engineer.

Potential Applications

Nonetheless, the CSAIL team believes PixelPlayer represents an important step toward more general audiovisual intelligence systems. Beyond its obvious uses in music education and production, it could enable a host of new applications at the intersection of sound and vision:

• Smart home devices that can localize the source of a suspicious noise
• AR/VR experiences with realistic spatial audio rendered from visual scenes
• Robots that use visual cues to better parse and respond to auditory commands
• Improved video conferencing with speaker-specific audio enhancement
• Safety systems that analyze road scenes to detect approaching sirens
• Audiovisual captioning tools for the hard of hearing

Moreover, the pixel-level precision of PixelPlayer‘s sound attribution could be transformative for sound design in games and movies. Imagine being able to click on any object in a video scene and adjust its sonic properties, or use computer vision techniques to automatically generate realistic foley sounds to match an animation.

On the research side, PixelPlayer provides a powerful new tool for music information retrieval (MIR) tasks like instrument detection, pitch tracking, and score transcription. By learning to align these multimodal features, it could help catalog and index large audio and video archives. It might even be able to reconstruct "silent videos" of musical performances by synthesizing the most likely sounds to match each player‘s physical motions.

Integration with Other Music AI Systems

Perhaps most intriguingly, PixelPlayer could be combined with other cutting-edge music AI models for novel creative workflows. Consider a system like Google Magenta‘s NSynth, which uses neural networks to generate new sounds by interpolating between different instruments.

What if you could use PixelPlayer to decompose a music video into isolated instrumental tracks, send those to NSynth to be morphed and recombined, then reintegrate the resulting audio back into the original video? You‘d have a powerful toolkit for creating remixes and mashups with both sonic and visual fidelity.

Or imagine using OpenAI‘s Jukebox model to generate new music in the style of an existing band, then having PixelPlayer provide a plausible visual accompaniment by finding and stitching together matching video clips of each instrumental part. The creative possibilities at the intersection of these audiovisual AI techniques are vast and largely unexplored.

Challenges and Future Directions

Of course, significant challenges remain in scaling up PixelPlayer to real-world complexity. The current model was trained primarily on videos of small musical ensembles in controlled studio settings. More work is needed to handle diverse, noisy environments with multiple sound sources and obstructed visuals.

Additionally, the computational requirements of running three CNNs in parallel are non-trivial, making real-time processing a challenge on consumer hardware. Optimizations and compression techniques will be needed to make PixelPlayer practical for everyday use.

Nonetheless, the core idea of using self-supervised learning to discover audiovisual correspondences is extremely powerful. As training datasets and architectures continue to evolve, we can expect to see more AI systems that can fluidly reason across sensory modalities in human-like ways.

PixelPlayer offers an exciting glimpse of this multimodal future. It represents a new class of AI tools that don‘t just analyze or generate in one mode, but deeply understand the interplay between the auditory and the visual. As such, it has the potential to not only change how we experience music, but to fundamentally reshape how we think about the relationship between sound and sight in our increasingly audiovisual world.