Building an AI-Powered App to Translate Spanish Speech to Nahuatl with Gradio and Hugging Face

Introduction

Language is a vital part of human culture and social connection. However, increased globalization and cultural assimilation have put many of the world‘s languages at risk. According to UNESCO, at least 40% of the 7,000+ languages spoken worldwide are endangered, with one disappearing every two weeks on average. When a language becomes extinct, we lose not only a means of communication but a wealth of cultural heritage, traditional knowledge, and unique ways of perceiving the world.

Thankfully, advances in artificial intelligence and machine learning are providing new ways to document, analyze and revitalize endangered languages. Tools like automatic speech recognition (ASR) and machine translation (MT) can rapidly accelerate the digitization of spoken and written language data. Apps powered by language AI models can help younger generations engage with ancestral tongues in more accessible, engaging formats.

In this tutorial, we‘ll walk through how to build a web app that transcribes spoken Spanish and translates it into the Nahuatl language, spoken by over 1.5 million people across Mexico. Our app will leverage open source libraries and pretrained models to achieve state-of-the-art accuracy while minimizing development time and compute cost. By the end, you‘ll have a working prototype you can adapt to other indigenous and endangered language pairs.

Tools & Technologies

To build our Spanish-to-Nahuatl speech translation app, we‘ll use the following open source tools and libraries:

• Gradio: A Python library for building web UIs for machine learning models with just a few lines of code. We‘ll use Gradio to create the frontend interface for uploading/recording speech and displaying translations.

• Hugging Face: An open source platform for sharing pretrained models, with an active community and thousands of models to choose from. We‘ll use Hugging Face to access pretrained Spanish ASR and Spanish-to-Nahuatl translation models, as well as to host our finished app.

• Wav2Vec2: A state-of-the-art speech recognition model pretrained on 960 hours of unlabeled speech in 50+ languages. We‘ll use a Wav2Vec2 model fine-tuned for Spanish ASR.

• T5: A powerful text-to-text transformer pretrained on a huge corpus of web pages, books, and Wikipedia articles. We‘ll use a T5 model fine-tuned to translate between Spanish and Nahuatl.

• Transformers: The Hugging Face library for working with pretrained transformer models for NLP and speech tasks. Provides high-level APIs for inference.

Implementation Steps

1. Set up Hugging Face account & repo

First create a free Hugging Face account if you don‘t already have one. From your account page, click "New Model" to create a repository for your app. Give it a name and select "Gradio" for the "App to use" option. This will generate starter code and configuration for deploying a Gradio app on Hugging Face‘s Spaces platform.

2. Install dependencies

In the generated requirements.txt file, add the following Python libraries needed for the app:

gradio
transformers 
torch
librosa

These will be automatically installed when the app is deployed.

3. Load pretrained models

Create a new app.py file in your repo. At the top, import the necessary libraries and load the pretrained ASR and translation models from the Hugging Face model hub:

import gradio as gr
import librosa
from transformers import pipeline

# Load Spanish ASR model
asr_model = "jonatasgrosman/wav2vec2-large-xlsr-53-spanish"
asr_pipeline = pipeline("automatic-speech-recognition", model=asr_model)

# Load Spanish-Nahuatl translation model  
mt_model = "fma-hispanic/mt5-small-spanish-nahuatl"
mt_pipeline = pipeline("translation_spanish_to_nahuatl", model=mt_model)

This loads a Wav2Vec2 model pretrained on Spanish speech data and a T5 model fine-tuned on Spanish-Nahuatl parallel text data. The pipeline function provides an easy interface for generating predictions from each model.

4. Define prediction function

Next, write a function that takes a Spanish speech recording as input, transcribes it to Spanish text with the ASR model, then translates the Spanish text to Nahuatl with the MT model:

def translate(audio):
    # Transcribe Spanish speech to text
    spanish_text = asr_pipeline(audio)["text"]

    # Translate Spanish text to Nahuatl
    nahuatl_text = mt_pipeline(spanish_text)[0]["translation_text"]

    return spanish_text, nahuatl_text

This function will be called each time a user submits an audio recording to the app.

5. Create Gradio interface

Finally, use Gradio‘s Interface class to create the web UI for the app, linking the prediction function to the appropriate input and output components:

ui = gr.Interface(
    fn=translate, 
    inputs=gr.Audio(source="microphone", type="filepath"), 
    outputs=[gr.Textbox(label="Spanish Transcription"), 
             gr.Textbox(label="Nahuatl Translation")],
    title="Spanish to Nahuatl Speech Translation",
    description="Record or upload a Spanish audio clip to transcribe it and translate it to Nahuatl.",
)

ui.launch()

This code creates a web UI with a microphone button to record Spanish speech, and two text output boxes to display the Spanish transcription and Nahuatl translation respectively. The launch() function starts the app server.

6. Test and debug

With all the code in place, commit the changes to your Hugging Face repo. The app will automatically build and deploy in a few minutes. Once it‘s live, test it out with some sample Spanish audio recordings. Debug any errors that occur by checking the logs.

Some potential issues to watch out for:

• CUDA out of memory errors from loading large models
• Slow inference times with long audio inputs
• Mistranslations or incorrect Nahuatl output for certain phrases
• Audio quality or background noise impacting ASR accuracy

Iteratively test and tweak the app until you have a minimum viable prototype. From there, you can work on optimizing performance, enhancing the UI/UX, and expanding functionality.

Challenges & Limitations

While our Spanish-to-Nahuatl app demonstrates the power of language AI to make indigenous tongues more accessible, it also highlights several key challenges:

Data Scarcity: High-quality ASR and MT models require large amounts of transcribed speech data and parallel translated sentence pairs to train on. For low-resource languages like Nahuatl, this data is often limited or nonexistent. As a result, models may struggle with certain dialects, topics, or use cases not represented in the training data.

Algorithmic Bias: If the training data is skewed toward certain demographics (e.g. age, gender, region), the resulting model predictions can be biased. This is especially problematic for indigenous language technology, as these communities have historically been underrepresented in the data collection process.

User Privacy: Collecting and using people‘s voices for ASR model development raises important questions around data protection, user consent, and intellectual property rights. Extra care must be taken to ensure that individuals‘ privacy is respected and that indigenous communities retain control over their linguistic data.

Model Limitations: While large pretrained models are highly accurate in general, they can still make mistakes, especially for out-of-domain audio or complex linguistic phenomena. Our app is not a substitute for human interpreters and should be used with caution for sensitive or high-stakes scenarios.

Despite these challenges, we believe apps like this can play a valuable role in making endangered languages more visible and viable in the digital age. By open sourcing our code and process, we hope to inspire more developers to experiment with language technology for social good.

Next Steps

There are many exciting ways this prototype could be extended and improved:

1. Expand to more languages: Fine-tune the ASR and MT models for additional endangered and indigenous language pairs. Allow users to select their desired input and output languages.

2. Enhance ASR robustness: Augment the training data with diverse speakers and recording environments to improve model performance on real-world speech. Experiment with noise reduction and speaker diarization techniques.

3. Add Nahuatl speech synthesis: Provide an option to generate spoken Nahuatl audio from the translated text using a text-to-speech (TTS) model. This could aid in pronunciation learning.

4. Improve user experience: Allow users to edit transcriptions to fix ASR errors. Provide a feedback mechanism to collect additional speech and translation data. Gamify the app with quizzes and challenges.

5. Deploy on mobile: Package the app for iOS and Android to enable offline, on-device speech translation. Optimize the model architectures for mobile CPU/GPU inference.

6. Engage indigenous communities: Partner with Nahuatl speakers and language preservation organizations for app development, data collection, and user testing. Ensure indigenous sovereignty and data governance.

We‘re excited to see where the open source community takes this prototype next. By combining cutting-edge AI with community-driven language revitalization efforts, we can help ensure that the world‘s linguistic diversity is preserved and promoted for generations to come.

Conclusion

In this post, we walked through how to build a web app that translates spoken Spanish to Nahuatl text using open source speech recognition and machine translation models. With just a few dozen lines of Python code and pre-trained models, we were able to achieve a high degree of accuracy on this challenging task.

However, our app is just a starting point. Overcoming the data scarcity, algorithmic bias, and deployment challenges faced by indigenous and endangered languages will require a concerted, interdisciplinary effort across AI/ML practitioners, app developers, linguists, and native speaker communities.

If you‘re inspired to contribute to this effort, we encourage you to experiment with fine-tuning language models for your ancestral or local tongues, or explore other domains like healthcare, education, or the arts where language technology can make an impact. Feel free to adapt and remix our code to suit your specific use case.

Together, we can build a more linguistically diverse and digitally equitable world, one app at a time. Qualli tonalli ihuan tlazohcamati!