# Generating Carnatic Music with AI and Google Magenta

- Canonical: https://33rdsquare.com/generate-carnatic-music-with-google-magenta/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Music is a universal language that has the power to transcend barriers and touch the depths of the human soul. Among the rich musical traditions of the world, the classical music of South India, known as Carnatic music, stands out for its complex melodic and rhythmic structures, and ability to evoke a wide range of emotions and moods.

In recent years, advances in artificial intelligence and machine learning have opened up exciting new possibilities for generating music computationally. Google‘s Magenta project is at the forefront of this field, providing powerful tools and models for working with musical note sequences. In this article, we‘ll explore how Magenta can be used to generate Carnatic music, diving into the core concepts and walking through code examples.

## The Elements of Carnatic Music

Before we jump into the technical details, let‘s briefly go over the fundamental elements that define the Carnatic music system:

**Ragas:** A raga is the melodic framework around which a composition is built. It consists of a set of notes (usually 5-7) with specific rules around which notes can be used when ascending or descending, as well as characteristic phrases and ornamentations. There are 72 basic ragas known as melakarta ragas, with thousands of additional ragas derived from them.

**Talas:** The tala provides the rhythmic structure for a composition. It consists of repeated cycles, each divided into beats with emphasis on certain parts of the cycle. There are 7 basic talas and 35 complex talas (jatis).

**Gamakas:** Gamakas are a core part of the Carnatic style and refer to the heavy use of ornamentations like vibrato and slides between notes. This gives the music its continuous, flowing quality.

**Shruti:** Carnatic music uses a system of microtonal pitches known as shrutis. While a typical Western scale divides an octave into 12 semitones, the Carnatic system considers 22 shrutis.

These elements come together in a Carnatic composition to create the distinct rich sound and emotional resonance the style is known for. While Carnatic and Western music do share the foundational 12 notes in an octave, these additional layers make the Carnatic system quite unique.

## Mapping Carnatic to MIDI

To represent Carnatic music digitally, we need a way to map the notes to a format the computer can understand. Fortunately, the ubiquitous MIDI (Musical Instrument Digital Interface) standard provides a great starting point.

In MIDI, notes are represented by numbers from 0-127, with each number corresponding to a specific pitch. By defining a mapping from Carnatic notes (S, R1, R2, G1, G2, M1, M2, P, D1, D2, N1, N2) to MIDI pitches, we can create a digital representation that captures the basic note information.

Here‘s an example mapping:

- S : p = 60
- R1 : p = 61
- R2 : p = 62
- G1 : p = 63
- G2 : p = 64
- M1 : p = 65
- M2 : p = 66
- P : p = 67
- D1 : p = 68
- D2 : p = 69
- N1 : p = 70
- N2 : p = 71

With this, we can now represent Carnatic notes and patterns in a format Google Magenta can work with.

## Intro to Magenta NoteSequences

At the core of Google Magenta is the NoteSequence representation – a protocol buffer format that encodes a series of notes, each with pitch, instrument, velocity, and start/end times. This provides a flexible way to represent any type of sequence-based music.

Magenta provides a high-level Python API to easily create, manipulate, and play back NoteSequences. Let‘s walk through a basic example of generating a simple Carnatic raga.

The mayamalavagowla raga is traditionally the first one taught to Carnatic music students. It has the following ascending and descending note sequences:

- Arohana (ascending): S R1 G2 M1 P D1 N2 S
- Avarohana (descending): S N2 D1 P M1 G2 R1 S

Here‘s how we can generate this in Magenta:

```
from magenta.music import constants
from magenta.music import sequences_lib
from magenta.music.protobuf import music_pb2

# Define mapping of Carnatic notes to MIDI pitches
note_map = {‘S‘: 60, ‘R1‘: 61, ‘R2‘: 62, ‘G1‘: 63, ‘G2‘: 64, ‘M1‘: 65,
            ‘M2‘: 66, ‘P‘: 67, ‘D1‘: 68, ‘D2‘: 69, ‘N1‘: 70, ‘N2‘: 71}

# Create NoteSequence
sequence = music_pb2.NoteSequence()

# Add notes to sequence
seq = []
for note_str in ‘S R1 G2 M1 P D1 N2 S N2 D1 P M1 G2 R1 S‘.split():
  note = sequence.notes.add()
  note.pitch = note_map[note_str]
  note.start_time = len(seq) * 0.5
  note.end_time = note.start_time + 0.5
  note.velocity = 80
  seq.append(note)

# Play the sequence
sequences_lib.play_sequence(sequence, synth=constants.DEFAULT_SYNTH)
```

The code above defines the pitch mapping, creates a new NoteSequence, and then iterates through the raga note names, adding each one to the sequence with a fixed duration and velocity. Finally, we can use Magenta‘s `play_sequence` helper to play back the generated sequence.

We can plot the resulting sequence to visualize it:

```
plot = sequences_lib.plot_sequence(sequence)
plot.show()
```

This gives us a simple plot showing the notes of the mayamalavagowla raga over time. While basic, this provides a foundation we can build on to create more complex musical sequences.

## Melody RNNs for Note Sequence Continuation

In addition to the NoteSequence representation, Magenta also provides a number of pre-trained models for generative music tasks. One of these is the Melody RNN model, which learns to continue a given NoteSequence in the style it was trained on.

We can use a Melody RNN to continue a Carnatic raga sequence and see what variations it produces. Here‘s an example:

```
from magenta.models.melody_rnn import melody_rnn_sequence_generator
from magenta.models.shared import sequence_generator_bundle

# Define input NoteSequence
input_sequence = music_pb2.NoteSequence()

# Add example raga notes
seq = []
for note_str in ‘S R1 G2 M1 P D1 N2‘.split():
  note = input_sequence.notes.add()
  note.pitch = note_map[note_str]
  note.start_time = len(seq) * 0.5
  note.end_time = note.start_time + 0.5
  note.velocity = 80
  seq.append(note)

# Load Melody RNN bundle
bundle = sequence_generator_bundle.read_bundle_file(‘/path/to/bundle.mag‘)

# Generate continuation
generator_map = melody_rnn_sequence_generator.get_generator_map()
melody_rnn = generator_map[‘basic_rnn‘](checkpoint=None, bundle=bundle)
num_steps = 32
temperature = 1.0

continued_sequence = melody_rnn.generate(input_sequence, num_steps, temperature)

# Play continuation
sequences_lib.play_sequence(continued_sequence, synth=constants.DEFAULT_SYNTH)
```

After defining our input sequence with the first few notes of a raga, we load a pre-trained Melody RNN from a bundle file. We then use the RNN to generate a continuation of the input sequence for a specified number of steps and "temperature" (randomness).

The continued sequence will start with our input raga notes and then generate new notes in a similar style. However, one challenge is that the model has no concept of the raga and may introduce notes outside of the valid set. Keeping the generated notes strictly within the raga is an open problem.

## Random Note Generation

As an alternative to the RNN approach, we can also generate valid raga sequences through random sampling of the notes. While less "intelligent", this brute-force method can ensure that we only generate notes belonging to the chosen raga.

Here‘s a simple example of how we could randomly generate a sequence for the kalyani raga, which uses the notes S, R2, G2, M1, P, D2, N2:

```
import random

kalyani_notes = ‘S R2 G2 M1 P D2 N2‘.split()

seq = []
for i in range(16):
  note_str = random.choice(kalyani_notes)
  note = input_sequence.notes.add()
  note.pitch = note_map[note_str]
  note.start_time = len(seq) * 0.5
  note.end_time = note.start_time + 0.5
  note.velocity = 80
  seq.append(note)

sequences_lib.play_sequence(input_sequence, synth=constants.DEFAULT_SYNTH)
```

This generates a random 16-note sequence using only the notes from the kalyani raga. While quite basic, this approach can be extended with more sophisticated sampling techniques that take into account common note patterns and phrases.

## The Future of AI-Generated Carnatic Music

The examples we‘ve covered here only scratch the surface of what‘s possible with AI music generation. In the time since the original article inspiring this post was written, there have been significant advances in the field.

New model architectures like Transformers have shown impressive results in capturing long-term dependencies and structure in sequence generation tasks. And end-to-end models like OpenAI‘s Jukebox are now able to generate full songs complete with instrumentation and vocals.

However, most of this work has been focused on Western music, and the unique aspects of Carnatic music mean that specialized models and representations are needed to fully capture its complexity and subtlety. Doing so could lead to powerful new tools for Carnatic musicians and students, aiding in composition, improvisation, and practice.

Some promising directions for future work include:

- Improved representations that can encode the continuous pitch movements (gamakas) and microtonal shrutis that are fundamental to the Carnatic style
- Models that are trained specifically on large datasets of Carnatic compositions and performances
- Interfaces and visualizations tailored to the needs and workflows of Carnatic musicians
- Intelligent practice tools that can generate variations of phrases and provide feedback on student performances

As AI continues to evolve and intersect with the world of music, the potential for new forms of creative expression and cultural exchange is vast. By bringing cutting-edge machine learning techniques to one of the world‘s great classical music traditions, we can both preserve and enrich this centuries-old art form while opening up new possibilities for its future.

---

Source: [Generating Carnatic Music with AI and Google Magenta](https://33rdsquare.com/generate-carnatic-music-with-google-magenta/)