Hacking the NVIDIA NeMo Script to Download Mozilla Common Voice Data
If you‘re interested in building speech recognition models, you‘ve likely come across NVIDIA NeMo. NeMo (Neural Modules) is a powerful open-source toolkit that makes it easy to develop state-of-the-art conversational AI models with just a few lines of code. One of its key features is end-to-end automatic speech recognition (ASR) model development.
To train robust ASR models, we need large datasets of transcribed speech. A great option is Mozilla‘s Common Voice, an open-source, multi-language dataset of voiced recordings. The latest release, version 10.0, contains a whopping 13,000 hours of validated speech data across 76 languages!
However, there‘s a catch. The English subset alone is over 500 GB! Downloading and preparing such a massive dataset is a major challenge. Thankfully, the NeMo team provided a handy Python script to automate the process. There‘s just one problem…it doesn‘t work anymore!
Don‘t worry though, we can fix it. In this post, I‘ll walk you through how I investigated the issues with the script, made the necessary changes, and got it working again. Consider this your unofficial guide to hacking the NeMo Common Voice downloader!
By the end, you‘ll be able to download and process the latest Common Voice data with ease. More importantly, you‘ll pick up some invaluable skills in reading and understanding unfamiliar code, making modifications, and modernizing older scripts.
Let‘s get started!
The Original Script
The script in question was originally written by SeanNaren and is available here in the NeMo GitHub repo. At a high level, here‘s what it‘s supposed to do:
- Download the specified Common Voice release and language, and untar it
- Convert the MP3 files to WAV format
- Process the clips to get metadata like duration
- Generate manifest files containing the filepaths, transcripts, and durations
However, if you try running it, you‘ll quickly run into issues. The problem is that the script hasn‘t been updated in a while, so it‘s out of sync with changes to how the Common Voice datasets are packaged and hosted.
Specifically, there are two main issues:
- The download URLs are outdated. Common Voice moved to a different hosting solution on AWS.
- The datasets now come as .tar.gz files instead of just .tar, which requires different extraction logic.
No worries though – these are fairly straightforward fixes. I‘ll take you through it step-by-step.
Fixing the Download Logic
The first order of business is getting the script to actually download the data again. If you look at the current hosting location for the datasets, you‘ll notice the URL structure looks like:
where {version} is something like "cv-corpus-10.0-2022-07-04" and {language} is the 2-letter locale code like "en".
In the original script, the URL was hardcoded, so let‘s update that:
COMMON_VOICE_URL = "https://mozilla-common-voice-datasets.s3.dualstack.us-west-2.amazonaws.com/{}/{}-{}.tar.gz".format(
args.version, args.version, args.language
)
I‘ve parameterized it to take the version and language from command line arguments. This will make the script more reusable if the URL changes again in the future.
While we‘re at it, let‘s also add logic to extract the .tar.gz file, since the original script was expecting just .tar:
tar = tarfile.open(target_file, "r:gz") tar.extractall(target_unpacked_dir) tar.close()
We open the tarfile in gzip mode, extract it all to the target directory, and close it. Easy enough!
Processing the Audio Files
With the correct dataset in hand, the next step is to process the individual audio clips. The Common Voice datasets come with a set of TSV files that map the MP3 filenames to their transcripts.
The script reads these TSVs, converts the MP3s to WAVs, and collects some metadata. Here‘s the original processing logic:
def process(x):
file_path, text = x
file_name = os.path.splitext(os.path.basename(file_path))[0]
text = text.lower().strip()
audio_path = os.path.join(audio_clips_path, file_path)
output_wav_path = os.path.join(wav_dir, file_name + ‘.wav‘)
tfm = Transformer()
tfm.rate(samplerate=args.sample_rate)
tfm.build(
input_filepath=audio_path,
output_filepath=output_wav_path
)
duration = sox.file_info.duration(output_wav_path)
return output_wav_path, duration, text
This looks reasonable. For each row in the TSV, it:
- Extracts the MP3 path and transcript
- Builds the input and output filepaths
- Converts the MP3 to WAV using SoX
- Gets the audio duration of the converted WAV
- Returns the WAV path, duration, and transcript text
The only change I‘d suggest is using FFmpeg instead of SoX for the conversion. It‘s a more widely used tool and doesn‘t require separate bindings. Here‘s what that looks like:
import ffmpeg
output_wav_path = os.path.join(wav_dir, file_name + ‘.wav‘)
stream = ffmpeg.input(audio_path)
stream = ffmpeg.output(stream, output_wav_path, acodec=‘pcm_s16le‘, ac=1, ar=str(args.sample_rate))
ffmpeg.run(stream, quiet=True, overwrite_output=True)
duration = sox.file_info.duration(output_wav_path)
We use ffmpeg-python to create an input stream from the MP3, set the output format to WAV with the specified sample rate, and run the conversion.
The rest of the processing logic remains the same. It‘s wrapped in a ThreadPool to parallelize the conversions across multiple CPUs.
Generating the Manifests
The final step is to take the processed audio paths, durations, and transcripts and assemble them into NeMo-compatible JSON manifest files. This allows NeMo to efficiently read the data for training.
Here‘s the relevant snippet:
def create_manifest(data, output_name, manifest_path):
output_file = Path(manifest_path) / output_name
output_file.parent.mkdir(exist_ok=True, parents=True)
with output_file.open(‘w‘) as f:
for wav_path, duration, text in data:
f.write(json.dumps({
‘audio_filepath‘: os.path.abspath(wav_path),
‘duration‘: duration,
‘text‘: text
}) + ‘\n‘)
For each row in the processed data, it creates a JSON line containing the absolute audio filepath, the duration in seconds, and the transcript text. These lines are written to the specified output file.
The only suggestion I have here is to use with to ensure the file handle is always closed properly. Other than that, it does the job!
Putting It All Together
Now that we‘ve covered the major parts of the script, let‘s see how they fit together in the main execution flow:
def main():
data_root = args.data_root
os.makedirs(data_root, exist_ok=True)
target_unpacked_dir = os.path.join(data_root, "CV_unpacked")
os.makedirs(target_unpacked_dir, exist_ok=True)
if not os.listdir(target_unpacked_dir):
# Data not found, download and extract
logging.info(f"Downloading and unpacking {args.version} {args.language} to {target_unpacked_dir}...")
filename = wget.download(COMMON_VOICE_URL, data_root)
target_file = os.path.join(data_root, os.path.basename(filename))
tar = tarfile.open(target_file, "r:gz")
tar.extractall(target_unpacked_dir)
tar.close()
else:
logging.info(f"Found existing unpacked dataset at {target_unpacked_dir}")
folder_path = os.path.join(target_unpacked_dir, args.version + f‘/{args.language}/‘)
for csv_file in args.files_to_process:
logging.info(f"Processing {csv_file}...")
data = process_files(
csv_file=os.path.join(folder_path, csv_file),
data_root=os.path.join(data_root, os.path.splitext(csv_file)[0]),
num_workers=args.num_workers
)
logging.info(f"Creating {csv_file} manifest...")
create_manifest(
data=data,
output_name=f‘commonvoice_{os.path.splitext(csv_file)[0]}_manifest.json‘,
manifest_path=args.manifest_dir
)
logging.info("Done!")
The flow is:
- Create the output directories if they don‘t exist
- Check if the dataset is already downloaded and extracted
- If not, download and untar it
- Loop through the specified input TSVs
- Process the rows in each TSV to convert audio and collect info
- Generate a manifest JSON for each TSV
- Log a completion message
I‘ve added some additional logging to make the progress clearer and tweaked the directory structure a bit. The main logic remains the same though.
Tips and Tricks
Before we wrap up, I want to share a few bonus tips I picked up while working on this:
-
Use the argparse module for command line parameters. It makes your script a lot more flexible and saves you from hardcoding paths.
-
Don‘t be afraid to refactor code into smaller functions. The original script had all the logic in one big main() function, which made it harder to follow. Breaking it up into process_files(), create_manifest(), etc. makes the flow much clearer.
-
Take advantage of parallelization where possible. Converting thousands of MP3s to WAV takes a while, so using a ThreadPool speeds it up considerably.
-
Always handle file creation and deletion gracefully. Use with blocks to ensure file handles are closed, and consider adding try/except blocks for more robust error handling.
-
Log liberally! Good logging makes it much easier to debug issues and track progress, especially for long-running scripts. The logging module makes it easy to adjust verbosity too.
-
If you get stuck on a script, don‘t hesitate to look for more recent forks or alternatives on GitHub. Chances are someone else has run into the same issue and may have already fixed it!
Final Thoughts
Well folks, that‘s a wrap! I hope this deep dive into updating the NeMo Common Voice download script has been educational and empowering.
While the specific fixes we covered may not apply directly to other projects, the general approach certainly does. Being able to read code written by others, understand its purpose, and modify it to fit your needs is an invaluable skill in this field.
I encourage you to take this script and experiment with it further. Try downloading other languages, or adapt it to work with other speech datasets. The more you practice digging into unfamiliar codebases, the more comfortable you‘ll become.
If you have any questions or suggestions for improving this post, please leave a comment below. I‘m always looking to learn from this awesome community. And if you found this guide helpful, I‘d really appreciate you sharing it with others who might benefit.
Until next time, happy hacking!
