Detecting COVID-19 From Cough Sounds Using AI and Mel Spectrograms

ByJordan Brown

The COVID-19 pandemic has had a profound global impact since emerging in late 2019. This highly contagious viral disease, caused by the SARS-CoV-2 coronavirus, has led to millions of infections and deaths worldwide. One of the key challenges in containing its spread is the difficulty in quickly identifying infected individuals, as symptoms may not appear for several days after exposure. Rapid, accessible screening methods could play an important role in early detection and preventing transmission.

A common symptom of COVID-19 is cough. While coughing alone is not definitive, as it is associated with many other conditions, the sound of a COVID-19 cough has certain distinct acoustic qualities. This opens up the possibility of using artificial intelligence to analyze cough sounds and screen for likely COVID-19 cases.

In this article, we‘ll explore how deep learning techniques like convolutional neural networks (CNNs) can be used to automatically detect COVID-19 from mel spectrogram representations of cough audio recordings. Mel spectrograms offer a visual way to represent the frequency content of audio signals over time, and have proven useful in analyzing speech and other sounds.

We‘ll go through the steps to build a CNN model that takes mel spectrogram images of cough sounds as input, and learns to classify them as either COVID-19 positive or negative. This will include:

  1. Obtaining an audio dataset of COVID-19 and healthy cough sounds
  2. Preprocessing the audio and converting into mel spectrograms
  3. Designing and training a CNN architecture on the spectrogram images
  4. Evaluating the model‘s predictive performance

By the end, you‘ll see how AI and audio processing techniques like mel spectrograms can potentially be used to develop assistive screening tools for COVID-19 and other respiratory diseases.

The Dataset

The first step is to obtain an appropriate dataset to train and evaluate the system. For this task, we‘ll use the Coswara dataset, which contains crowd-sourced audio recordings of cough sounds from COVID-19 positive and healthy individuals. Coswara is an ongoing data collection project aimed at gathering respiratory audio samples to aid in diagnosing COVID-19.

The dataset includes around 1500 cough recordings in total, with about 300 from COVID-positive subjects. While this dataset is relatively small, it provides a good starting point to investigate the potential of this approach. In a real deployment, it would be important to collect a larger and more diverse dataset to improve robustness and minimize bias.

The audio files are provided in .wav format with a 44.1 kHz sample rate. Before using the raw audio, some preprocessing is needed.

Generating Mel Spectrograms

To convert the raw cough audio into a suitable representation for CNN classification, we‘ll compute mel spectrograms. A spectrogram is a 2D visual representation that shows how the frequency content of a signal varies over time. The horizontal axis represents time, while the vertical axis represents frequency. Intensity at each time-frequency point is indicated by color or brightness.

Mel spectrograms are a special type of spectrogram where the frequency axis is converted to a mel scale. The mel scale is based on human perception of pitch – listeners are more attuned to differences at lower frequencies than higher ones. Using a mel frequency scale makes the spectrogram more closely match how humans hear sound.

Here‘s the basic process to generate mel spectrograms from the cough recordings:

  1. Load the .wav audio file
  2. Resample the audio to a desired sample rate (e.g. 22050 Hz) if needed
  3. Compute the linear spectrogram using the short-time Fourier transform (STFT)
    • The STFT divides the signal into overlapping frames and applies a window function (e.g. Hann window)
    • Taking the discrete Fourier transform (DFT) of each frame yields the spectrum over time
  4. Convert the frequency axis from Hz to mels using a mel filterbank
    • The filterbank sums frequency components according to a mel-spaced set of triangular filters
  5. Take the log of the mel spectrogram values to compress the dynamic range
  6. Normalize the spectrogram values to [0, 1]
  7. Save the mel spectrogram as an image file

This process can be easily implemented in Python using libraries like Librosa, NumPy, and Matplotlib. Here‘s some example code to generate a mel spectrogram from a .wav file:

import librosa 
import librosa.display
import numpy as np
import matplotlib.pyplot as plt

# Load the audio file
audio, sample_rate = librosa.load(‘cough.wav‘, res_type=‘kaiser_fast‘) 

# Compute the mel spectrogram
mel_spec = librosa.feature.melspectrogram(y=audio, sr=sample_rate)
mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)

# Display the spectrogram
librosa.display.specshow(mel_spec_db, x_axis=‘time‘, y_axis=‘mel‘, sr=sample_rate)
plt.colorbar(format=‘%+2.0f dB‘)
plt.show()

After generating mel spectrogram images for each cough sound in the dataset, we‘re ready to design the CNN model.

CNN Architecture

Convolutional neural networks have led to many breakthrough results in image classification over the past decade. Their ability to automatically learn hierarchical features from raw pixel data makes them well-suited for our task of recognizing patterns in mel spectrograms.

A basic CNN architecture for mel spectrogram classification could consist of the following:

  • Convolutional layers: These apply learned filters to extract features from the input image. Each filter activates in response to certain visual patterns. Multiple filters are used to build a rich representation. Typically, the first conv layers detect simple low-level patterns like edges and textures, while deeper layers combine these to recognize more complex high-level features.

  • Pooling layers: Max pooling is commonly used to progressively reduce spatial resolution as depth increases. This helps the network become invariant to small translations and distortions.

  • Dense layers: After the conv layers, the 2D feature maps are flattened to a 1D vector and passed through one or more fully-connected layers to perform classification or regression. The final layer outputs the predicted probability of each class.

Here‘s an example CNN architecture specified using the Keras API in TensorFlow:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout

model = Sequential([
    Conv2D(32, (3,3), activation=‘relu‘, input_shape=(64, 64, 3)), 
    MaxPooling2D((2, 2)),

    Conv2D(64, (3,3), activation=‘relu‘),
    MaxPooling2D((2, 2)),

    Conv2D(128, (3,3), activation=‘relu‘),
    MaxPooling2D((2, 2)),

    Conv2D(256, (3,3), activation=‘relu‘),
    MaxPooling2D((2, 2)),

    Flatten(),
    Dropout(0.5),
    Dense(512, activation=‘relu‘),
    Dense(1, activation=‘sigmoid‘)
])

This model takes 64×64 RGB images as input and applies four conv layers with increasing filter depth, each followed by max pooling. After the conv base, the feature maps are flattened and passed through two dense layers to make the COVID-19 probability prediction.

Dropout is used between the dense layers to reduce overfitting. The sigmoid output represents the probability of the input cough being COVID-positive (closer to 1) or negative (closer to 0). Binary cross-entropy is an appropriate loss function for this kind of binary classification.

Training

Before training the CNN, the mel spectrogram images need to be loaded and preprocessed. The Keras ImageDataGenerator class provides a convenient way to load batches of images from disk and apply real-time augmentation.

Augmentations like random scaling, cropping, and horizontal flipping are often used to synthetically expand the training dataset and improve the model‘s ability to generalize. For mel spectrograms, we could experiment with augmentations like time shifting, pitchshifting, and adding background noise.

Here‘s an example of creating train and validation generators:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(rescale=1./255, 
                                   horizontal_flip=True,
                                   vertical_flip=True)

train_generator = train_datagen.flow_from_directory(train_dir,
                                                    target_size=(64, 64), 
                                                    color_mode=‘rgb‘,
                                                    batch_size=32,
                                                    class_mode=‘binary‘) 

val_datagen = ImageDataGenerator(rescale=1./255)

val_generator = val_datagen.flow_from_directory(val_dir,
                                                target_size=(64, 64),
                                                color_mode=‘rgb‘,
                                                batch_size=32,
                                                class_mode=‘binary‘)

With the data generators ready, we can compile and train the CNN:

model.compile(optimizer=‘adam‘,
              loss=‘binary_crossentropy‘,
              metrics=[‘accuracy‘])

epochs = 50
history = model.fit_generator(train_generator,
                              steps_per_epoch=train_generator.samples // batch_size,
                              epochs=epochs,
                              validation_data=val_generator,
                              validation_steps=val_generator.samples // batch_size)

This trains the model for 50 epochs using the Adam optimizer to minimize binary cross-entropy loss. The model accuracy on the validation set is also monitored during training.

After training, the model weights can be saved to disk for future inference:

model.save(‘cough_covid_detector.h5‘)

Evaluation

To assess the trained model‘s predictive performance, we can evaluate it on a held-out test set of cough recordings. The test generator can be created similarly to the train and validation generators, but without any augmentation:

test_datagen = ImageDataGenerator(rescale=1./255)

test_generator = test_datagen.flow_from_directory(test_dir,
                                                  target_size=(64, 64),
                                                  color_mode=‘rgb‘,
                                                  batch_size=32,
                                                  class_mode=‘binary‘)

test_loss, test_acc = model.evaluate_generator(test_generator)
print(f‘Test accuracy: {test_acc:.3f}‘)

In addition to overall accuracy, it‘s important to consider metrics like sensitivity (true positive rate) and specificity (true negative rate), as the cost of false negatives and false positives may be quite different for a screening task.

A confusion matrix can give a more complete picture of the model‘s performance:

from sklearn.metrics import confusion_matrix

test_preds = model.predict_generator(test_generator)
test_preds = np.round(test_preds).flatten()
test_labels = test_generator.classes

cm = confusion_matrix(test_labels, test_preds)
tn, fp, fn, tp = cm.ravel()

print(‘Confusion Matrix‘)
print(cm)

sensitivity = tp / (tp + fn)
specificity = tn / (tn + fp)
print(f‘Sensitivity: {sensitivity:.3f}‘) 
print(f‘Specificity: {specificity:.3f}‘)

Visualizing the confusion matrix can also help identify if the model is biased toward one class over the other.

Conclusion and Future Work

In this article, we explored how convolutional neural networks can be applied to mel spectrogram representations of cough audio to automatically detect COVID-19. While this approach shows promise, there are many challenges to overcome before such AI tools could be deployed in practice.

Some key areas for future work include:

  • Collecting larger and more diverse datasets, with subject demographics and recording environments that match the intended use case. Dataset bias is a major concern for any machine learning application.

  • Experimenting with different mel spectrogram parameters (e.g. window size, hop length, mel bands) and CNN architectures to optimize predictive performance. Transfer learning from models pretrained on other audio classification tasks could also help.

  • Developing more sophisticated data augmentation techniques to improve model robustness, such as mixing in background noise, random frequency masking, and simulating microphone and room acoustics.

  • Investigating explanatory AI techniques to help interpret what features the CNN is using to make its predictions and identify potential failure modes. Visualization methods like class activation mapping and saliency maps may be useful here.

  • Integrating the cough sound analysis with other relevant data streams, like symptoms and risk factors, for a more comprehensive screening tool. Smartwatch apps that can continuously monitor for cough events could also extend this approach.

  • Conducting prospective clinical studies to validate the effectiveness of cough-based COVID-19 screening and compare with standard diagnostic methods like RT-PCR testing. Any screening tool must be evaluated in terms of real-world health outcomes.

There are also important ethical implications to consider around privacy, consent, fairness, and potential misuse. Transparency and input from diverse stakeholders will be critical in developing these technologies responsibly.

Audio AI has the potential to augment our response to COVID-19 and future infectious diseases, but significant work remains to translate research into reliable, effective, and equitable screening tools. As the pandemic evolves, it‘s crucial that we continue to explore innovative solutions while prioritizing safety and ethics.

If you found this article interesting, you can find the complete code examples on GitHub. You may also enjoy the following resources:

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts