Data Science Composes a New Future for Music Industry
The field of data science, with its power to extract insights and knowledge from vast quantities of information, is orchestrating a dramatic transformation across industries. And now, it‘s striking a chord in one of the most emotive and creative domains of human culture: music.
By treating music as a rich source of data, encompassing everything from audio features to listener behavior, data scientists are uncovering new ways to analyze, generate, and interact with musical content. These innovations are not only reshaping how music is created and consumed, but also redefining the skills and tools needed to succeed in the industry.
The Data-Driven Rhythms of Music Consumption
The digital transformation of the music business has been driven in large part by the rise of streaming platforms like Spotify, Apple Music, and Pandora. These services have not only changed how listeners access music, but also generated enormous troves of data on consumption patterns.
Consider that Spotify alone has over 350 million monthly active users across 178 markets, streaming over 70 million tracks. Every day, these users generate billions of data points around what, when, and how they listen. This behavioral data has become the heartbeat driving music streaming‘s growth, which now accounts for over 80% of industry revenues in the US [1].
Streaming services leverage this data to power recommendation systems that help listeners discover new music tailored to their tastes. By analyzing a range of factors, from explicit signals like plays and likes to implicit ones like skip rates and playlist additions, these systems can surface personalized suggestions that keep listeners engaged.
For example, Spotify‘s Discover Weekly playlist, which reached 40 million people in its first year, uses collaborative filtering to identify songs enjoyed by listeners with similar tastes [2]. This has proven a powerful tool for exposing users to new artists and genres, with 8,000 artists seeing over half their listeners come from Spotify‘s algorithmic playlists [3].
But beyond individual recommendations, streaming data is also giving the industry new insights into broader consumption trends that can inform decisions around marketing, A&R, touring and more. Spotify for Artists and Apple Music for Artists are among the analytics tools aimed at leveraging streaming data to help music creators grow their audiences and understand their impact.
Generative AI Joins the Jam Session
Data science isn‘t just helping recommend and promote music – it‘s also being used to create it from scratch. Recent breakthroughs in deep learning and generative modeling have given rise to AI systems capable of composing original music, even imitating specific styles and artists.
One of the key techniques powering this trend is the use of deep neural networks, which can learn hierarchical representations of musical data such as melodies, harmonies and rhythms. By training on large datasets of symbolic music (e.g. MIDI) or raw audio, these models can capture the patterns and structure of music composition and generate new works based on that understanding.
For example, OpenAI‘s Jukebox model, trained on 1.2 million songs across a variety of genres, can generate music samples in the style of popular artists like Katy Perry, Elvis Presley and Nas [4]. The model uses a hierarchical VQ-VAE architecture to compress audio into discrete representations at multiple levels of abstraction, from high-level "style tokens" down to low-level audio samples.
Meanwhile, Google‘s Magenta project has developed a range of generative models for both symbolic and audio music generation, such as MusicVAE for interpolating between melodic sequences and GANSynth for synthesizing individual instrument sounds [5]. These models leverage techniques like variational autoencoders and generative adversarial networks to learn compact latent representations of musical data.
Beyond tech giants, a number of startups are also commercializing generative AI for music, particularly for use cases like soundtrack creation. Amper Music and Aiva are among the companies offering AI-powered tools to generate customized instrumental tracks on-demand for content like videos, podcasts, and games.
Generative models are even finding their way into the toolkits of adventurous musicians and producers. Dadabots, a research group run by two musicians/technologists, has released albums of black metal and math rock generated entirely by neural nets trained on their own compositions [6]. Holly Herndon‘s 2019 album Proto features an "AI baby" named Spawn trained on the artist‘s voice [7].
While these creative applications are still nascent, they point towards a future in which data science augments and complements human musical creativity. As the field progresses, generative tools may become an essential part of the 21st century composer‘s repertoire.
Cracking the Code of Musical Understanding
Alongside generation, another key area where data science is making waves in music is the field of Music Information Retrieval (MIR). MIR encompasses a range of techniques for extracting meaningful information from musical data, whether symbolic formats like MIDI and sheet music or audio signals.
Some common MIR tasks include:
- Genre classification: Automatically categorizing music by genre based on audio features and/or metadata
- Chord recognition: Detecting the chord progressions present in a piece of music from an audio recording
- Emotion detection: Analyzing musical and acoustic features to predict the emotional valence (positive vs negative) and arousal (calm vs excited) of a song
- Audio source separation: Isolating individual instruments or vocal parts from a mixed audio track
- Automatic music transcription: Converting an audio recording into a symbolic format (e.g. sheet music) by detecting pitch, rhythm, etc.
MIR draws heavily on signal processing and machine learning techniques to model the complex hierarchical and temporal structure of music. Convolutional and recurrent neural network architectures have proven particularly effective at learning representations of musical audio that capture relevant features and relationships [8].
These techniques are not only academically interesting, but have important applications across the music industry value chain. For example:
- Music discovery and recommendation systems can use content-based features extracted by MIR to find songs that are musically similar, even if they lack collaborative filtering data
- Intelligent audio production tools can use MIR to automatically detect and correct problems like intonation errors or sibilance in vocal recordings
- Music education software can use MIR to provide learners real-time feedback on their playing or to automatically generate practice exercises matched to their skill level
- Musicological research can leverage MIR to quantitatively study elements like melodic patterns, rhythmic structures, and harmonic progressions across large musical corpora
By allowing machines to understand musical information in a more human-like way, MIR is both a key building block for music AI systems and a powerful tool for music industry professionals to enhance their work.
Conducting the Market and Research Symphony
The potential for data science to transform the music business is attracting significant interest and investment. The global market for AI in music is expected to grow from $329 million in 2019 to $2.79 billion by 2024, at a CAGR of over 50% [9].
This growth is being driven by a range of industry players, from major labels and streaming services to music tech startups. Some key industry trends and developments include:
- Sony‘s CSL research lab launching FlowMachines, an AI music composition tool, and signing a record deal for its AI-generated songs
- Spotify acquiring music intelligence startup Echo Nest and generative music startup Niland to bolster its discovery and personalization capabilities
- Abbey Road Studios launching an incubator for music tech startups leveraging AI and machine learning
- Increased venture funding for music AI startups, such as $5.5M for Amper Music, $3.6M for Popmorphic, $5M for AIMI, and $2.5M for Vochlea
On the research front, institutions around the world are pushing the boundaries of data science in music through groundbreaking projects and interdisciplinary collaborations. In addition to the MIDAS projects mentioned earlier, other notable work includes:
- MAESTRO, a dataset from Google of over 200 hours of virtuosic piano performances, captured with fine-grained MIDI data aligned to audio [10]
- Deep Learning for Music workshop at ISMIR 2020, exploring state-of-the-art techniques for audio classification, music generation, MIR, and more [11]
- FuturePulse, a European project using predictive analytics on streaming and social data to forecast emerging trends and talent in the music industry [12]
- AMAAI Lab at KAIST in Korea, studying deep learning for audio generation, voice synthesis, music classification and tagging
- Jukedeck Research, which developed pioneering work in neural audio synthesis and symbolic music generation before being acquired by TikTok parent company ByteDance
As research continues to advance the state-of-the-art in music AI, we can expect to see more of these innovations make their way into industry products and creative workflows in the coming years. However, realizing the full potential of data science in music will require ongoing collaboration between researchers, engineers, music industry professionals, and artists.
Tuning the Ethical Strings
As with any application of AI, the use of data science in music also raises important ethical considerations. Some key issues include:
- Copyright and IP: As generative models improve at mimicking artists‘ styles, there are open questions around ownership and royalties for AI-generated music. Should training data be considered fair use? Can AI imitate an artist without permission?
- Transparency and explainability: Complex AI systems can be black boxes, making it hard to understand how a music recommendation or generation system arrived at its outputs. This lack of transparency could hide potential biases or failure modes.
- Data bias and diversity: Music datasets used to train AI may underrepresent certain genres, cultures, or demographics. This can result in biased systems that reinforce mainstream tastes and disadvantage groups already marginalized by the industry.
- Artistic authenticity and authorship: Some worry that the use of AI in music could devalue the role of human creativity and lead to homogenization. Will listeners care about the "soul" behind the music? How to properly attribute human and machine contributions?
Navigating these challenges will require proactive governance frameworks and multi-stakeholder dialogue. Initiatives like the Open Music Initiative and the Fair Trade Music project are working to develop ethical data practices and ensure that the benefits of music AI are distributed equitably [13].
Technical approaches like differential privacy, federated learning, and secure multi-party computation could also help enable data-driven innovation while protecting user privacy [14]. And the development of more interpretable AI techniques could help build trust by explaining a music AI system‘s outputs.
As data science rewrites the rules of the music business, it will be crucial that social responsibilities are addressed alongside technological capabilities. Ultimately, the goal should be to support a sustainable, diverse music ecosystem in which data and AI are tools for empowerment, not extraction or control.
The Coda
Music has always been a deeply human art form, capable of stirring our emotions, connecting us to others, and expressing the ineffable. As we enter an era in which data and computation increasingly mediate our interactions with music, it‘s critical that we consider how to harness these new technologies in service of musical expression and experience.
Data science has immense potential to enrich and expand the possibilities of music, from helping artists and labels better understand and grow their audiences, to offering new creative tools and sparking new forms of interactive, adaptive musical media.
At the same time, the music industry and research community must grapple with complex questions around the aesthetic, economic, and cultural implications of ceding more control to algorithms. Engaging critically and proactively with these issues will be essential to ensuring that the data-driven future of music is one that reflects our values and ideals.
As data science continues to conduct a sea change across the music industry, one thing is clear: the ultimate metric of success should not be optimizing streams or squeezing out inefficiencies, but deepening our collective capacity for musical expression and experience. In a world overflowing with data, the songs that endure will still be those that speak to our common humanity – no matter who, or what, wrote them.
