Multilingual Text-to-Speech Models for Indic Languages

Introduction

India is home to astounding linguistic diversity, with 22 official languages and over 19,500 dialects spoken across the country. Creating text-to-speech (TTS) systems that can generate natural-sounding speech for even a fraction of these languages is a daunting challenge. In recent years, advances in deep learning and the development of multilingual TTS models have opened up exciting possibilities to support many Indic languages with a single model.

In this article, we‘ll take a deep dive into cutting-edge multilingual TTS models for Indic languages. We‘ll explore different approaches to building these models, evaluate the performance of state-of-the-art systems, and discuss remaining challenges and areas for future work. Whether you‘re an engineer interested in using Indic TTS models in your application or a researcher keen to advance the field, this article will provide a comprehensive overview of the latest developments.

How Multilingual TTS Models Work

Traditional TTS models are trained on a single language, learning to map from that language‘s writing system to speech audio. To support multiple languages, you would need to train a separate model for each language – an approach that quickly becomes impractical when dealing with India‘s numerous languages.

Multilingual TTS models aim to solve this by learning to generate speech for multiple languages with a single model. There are two main ways this can be achieved:

1. Combining language alphabets: The model uses a shared alphabet consisting of the union of characters across all supported languages. Language-specific embeddings are used to signal the target language during training and inference.

2. Using a common romanization scheme: Each language‘s native script is first transliterated into a common romanization like the International Alphabet of Sanskrit Transliteration (IAST). The model then learns to generate speech from this romanized representation.

In addition to language embeddings, multilingual TTS models often incorporate speaker embeddings to enable generation of speech in different voices. By conditioning on both a language and a speaker embedding, the model can be guided to generate speech in a particular language and voice at inference time.

Under the hood, multilingual TTS models typically consist of two main components:

• Acoustic model: Generates a Mel spectrogram (a time-frequency representation of speech) from a phoneme or character sequence. Usually implemented as an encoder-decoder architecture using recurrent neural networks (RNNs), transformers, or a combination.

• Vocoder: Converts the Mel spectrogram into an audio waveform. Popular approaches include WaveNet, WaveRNN, and WaveGlow. More efficient neural vocoders like LPCNet and HiFi-GAN have been proposed recently.

By sharing parameters across languages in the acoustic model and vocoder, multilingual models enable transfer learning and require less training data per language than monolingual models. Now let‘s look at some state-of-the-art multilingual TTS models for Indic languages.

Cutting-Edge Indic TTS Models

Silero Models

Silero Models is an open-source collection of pre-trained TTS models supporting many languages, including 9 Indic languages: Hindi, Malayalam, Manipuri, Bengali, Rajasthani, Tamil, Telugu, Gujarati, and Kannada. Their latest multilingual Indic TTS model (v3) has the following key features:

• Leverages aksharamukha, an open-source tool for Indic script conversions, to romanize input text using the ISO 15919 standard
• Jointly trained on 17 speakers across the supported languages
• Achieves natural-sounding speech while being lightweight enough to run in real-time on a CPU
• Accepts SSML tags for controlling aspects like voice, pitch, rate and prosody
• Can transfer voices across languages, e.g. generate Malayalam speech in a Hindi speaker‘s voice

To use the model, simply install the silero Python package and load the pre-trained model from torch.hub:

import torch
from aksharamukha import transliterate

model, example_text = torch.hub.load(repo_or_dir=‘snakers4/silero-models‘,
model=‘silero_tts‘,
language=‘indic‘,
speaker=‘v3_indic‘)

orig_text = "प्रसिद्द कबीर अध्येता, पुरुषोत्तम अग्रवाल का यह शोध आलेख, उस रामानंद की खोज करता है"

roman_text = transliterate.process(‘Devanagari‘, ‘ISO‘, orig_text)
print(roman_text)

audio = model.apply_tts(roman_text, speaker=‘hindi_male‘)

The resulting audio will be returned as a PyTorch tensor. You can easily convert it to a numpy array or save it to an audio file. Here are some audio samples showing the model‘s output for different languages:

As you can hear, the model produces fairly natural-sounding speech in all supported languages. There are some errors in pacing and pronunciation, likely stemming from the model‘s romanized input representation and imperfect alignment between languages. Nonetheless, it sets a strong baseline for open-source Indic TTS.

Considering the model supports 9 languages and 17 speakers in just 50 MB, and can run in real-time on 1-4 CPU threads, it is quite efficient. See the table below for average inference speeds at different output sample rates.

Google/Microsoft/Amazon TTS for Indic Languages

Google, Microsoft, and Amazon all provide commercial Indic TTS through their respective Cloud platforms. Let‘s compare their offerings:

Google Text-to-Speech covers 12 Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Sinhala, Tamil, Telugu, and Urdu. It supports both WaveNet and neural WaveRNN vocoders.

Microsoft Azure Speech Services supports 6 Indic languages: Bengali, Gujarati, Hindi, Malayalam, Tamil, and Telugu. Each language has 1-2 neural voices.

Amazon Polly offers neural TTS for Hindi and Indian English. It recently added two Indian English voices, Aditi and Raveena, which use a more advanced neural engine.

In general, these commercial systems produce higher quality output than open-source models like Silero due to more sophisticated model architectures and extensive hyperparameter tuning. They also undergo more thorough quality assurance testing. However, their usage is metered and can become expensive at scale. They also tend to support fewer speakers per language compared to open-source models.

Here are some samples of Google and Microsoft‘s Hindi TTS:

Indian Academia and Industry

India has a vibrant research community working on TTS systems for Indic languages. Notable open-source TTS projects from Indian academia and industry include:

• IndicTTS: A deep learning framework for building Indian language speech synthesis systems, with support for 13 languages. Developed at IIT Madras.

• Shree TTS: A concatenative TTS system for 13 Indian languages, based on the Festival framework. Built at IIIT Hyderabad.

• Vakyansh TTS: An open-source, deep learning based TTS toolkit supporting 4 Indic languages. Created by Swecha, a non-profit supporting free software.

While these systems are less fully-featured than commercial offerings, they are helping to democratize Indic TTS technology. By open-sourcing their code and datasets, they enable students and researchers to easily experiment with building TTS models.

Challenges and Future Directions

Despite the impressive progress in multilingual Indic TTS over the past few years, there are still many challenges to overcome. Here are some key issues and areas for future work:

1. Improving speech naturalness and expressiveness: Even state-of-the-art TTS systems produce speech that sounds somewhat robotic and lacks the full range of human prosody. We need models that can better capture the nuances of emotional speech. Recent advances like Tacotron 2 and FastPitch are promising steps in this direction.

2. Expanding language coverage: Current multilingual models only scratch the surface of India‘s linguistic diversity. We need to collect more training data and develop models that can generate speech for hundreds of under-resourced languages and dialects.

3. Few-shot adaptation: It is impractical to collect hours of high-quality recordings for every language. Adapting an existing multilingual model to a new language using just a few minutes of speech would greatly accelerate support for India‘s long tail of languages.

4. Faster and smaller models: On-device TTS has advantages for latency, cost, and privacy. But edge devices have limited compute and memory. Distilling large TTS models into compact student networks is an active area of research.

5. Improved romanization: Using a single romanization scheme across diverse scripts has drawbacks. A more nuanced mapping considering each language‘s phonology could yield better results. Alternatively, models that directly generate speech from native scripts are worth exploring.

6. Evaluation metrics: The gold standard for assessing TTS quality is subjective listening tests, which are time-consuming and expensive. Developing objective metrics that reliably predict human judgments of speech naturalness could accelerate research.

Multilingual TTS models will play a key role in breaking down language barriers and making information accessible to all Indians. While today‘s systems are still far from human parity, they demonstrate the feasibility of supporting India‘s many languages with a unified approach. With continued research and investment, we can look forward to a future where every Indian can access natural-sounding speech technology in their native language.

Conclusion

In this article, we surveyed the landscape of multilingual TTS models for Indic languages. We explained the key concepts behind these models, including different input representations and model architectures. We evaluated state-of-the-art open-source and commercial systems, comparing their performance and efficiency. Finally, we discussed outstanding challenges and promising directions for future research.

Multilingual Indic TTS is a rapidly evolving field with potential for major social impact. By open-sourcing code and sharing knowledge, we can accelerate progress and ensure this technology benefits all Indians. I encourage you to experiment with the models and resources linked in this article. What ideas do you have for improving Indic TTS? Feel free to share your thoughts in the comments below!