Automatically Generate Sports Highlights using Python and Speech Analysis

Have you ever wanted to quickly view the most exciting parts of a long sports match? Manually scrubbing through hours of video to find the key moments can be a painstaking process. Luckily, with some clever Python code, it‘s possible to automatically detect and extract the highlights to generate a concise summary of the best bits!

In this article, you‘ll learn a simple yet effective approach to automatically generate sports highlights from full match videos using basic speech analysis techniques. And the best part? It doesn‘t require any machine learning or computer vision knowledge!

By analyzing patterns in the audio commentary, we can identify the most exciting moments like goals, shots, and fouls based on the rise in volume and intensity of the commentator‘s voice. When a significant event occurs, commentators tend to raise their voice and speak faster and louder. We can detect these audio changes with some signal processing in Python.

Whether you want to quickly catch up on a game you missed, or create highlights for your own sports videos, this technique is a great way to do it with minimal coding. Let‘s dive in!

Overview of Highlight Generation Approaches

There are a few common ways to automatically generate highlights from sports videos:

Manual editing – An editor scrubs through the full video and manually cuts together the most interesting clips. This creates high-quality highlights but is very time-consuming.

Computer vision – CV techniques like object detection and action recognition can visually detect key moments like goals, saves, fouls etc. by analyzing the video frames. This requires substantial ML expertise to get good results.

Crowdsourcing – Fans vote on their favorite parts of the match, and the most popular clips are combined into a highlight reel. This can surface great moments but requires an active viewing audience.

Speech analysis – This approach analyzes patterns in the audio commentary to detect exciting moments from the intensity of the commentator‘s voice. It‘s fast, simple and doesn‘t require machine learning. We‘ll use this approach in this post!

Each method has pros and cons in terms of implementation complexity, result quality and generality. In this article we‘ll focus on speech analysis as a great way to get started with Python-based highlight generation without needing ML experience.

Audio Signal Processing Basics

To understand the speech analysis approach, we need to cover a few basic signal processing concepts.

An audio signal represents a sound, such as speech or music, in a digital format. The signal stores the amplitude (loudness) of the sound wave at each moment in time. We can plot the signal as a waveform showing amplitude over time:

Key characteristics of an audio signal include:

• Sampling rate – how many amplitude measurements are taken per second (e.g. 44.1 kHz)
• Bit depth – how many bits are used to store each amplitude value (e.g. 16-bit)
• Number of channels – mono (1), stereo (2) etc.

The amplitude of the waveform corresponds to the loudness of the audio. So by analyzing the change in amplitude, we can detect when the sound gets louder or quieter.

One way to characterize the amplitude is using the short-time energy. This measures the total amplitude in a short sliding window (e.g. 50ms) of the audio. The short-time energy will be higher for louder sounds:

By calculating the short-time energy at each window, we get an idea of how the audio loudness changes over time. Sudden large increases suggest an exciting moment!

The Speech Analysis Highlight Generation Algorithm

Now that we understand how audio signals work, let‘s see how we can use speech analysis to extract highlights:

1. Extract the audio track from the video file
2. Split the audio into short overlapping windows (e.g. 50ms)
3. Calculate the short-time energy for each window
4. Find windows where the energy exceeds some threshold
5. Extract the video clips corresponding to those windows
6. Merge the clips together into one highlight video

The idea is that the exciting moments in the video will have a spike in the commentary audio intensity, which we can detect from the short-time energy. We then grab the corresponding video clips containing those moments.

Let‘s see how to implement this in Python! We‘ll use the excellent MoviePy library to handle the audio and video processing.

Implementing the Speech Analysis Algorithm in Python

First, install MoviePy and the other required libraries:

pip install moviepy numpy matplotlib

Now let‘s import them:

from moviepy.editor import *
import numpy as np
import matplotlib.pyplot as plt

Load the video file and extract the audio:

video = VideoFileClip("match.mp4")
audio = video.audio

Split the audio into short windows and calculate the energy:

duration = audio.duration
sample_rate = audio.fps
window_size = 0.05 # 50ms
window_samples = int(window_size * sample_rate)
windows = []
for start in np.arange(0, duration, window_size):
    end = start + window_size
    if end >= duration:
        break
    window = audio.subclip(start, end)
    window_energy = np.sum(window.to_soundarray()**2)
    windows.append({"start": start, "end": end, "energy": window_energy})

We first calculate the duration and sample rate of the audio. This tells us how long it is and how many audio samples there are per second.

We then split the audio into short 50ms windows with a sliding frame, calculating the total energy for each window. We store the start time, end time and energy for each window in a list.

Let‘s plot the short-time energy for the whole audio:

window_times = [w["start"] for w in windows]
window_energies = [w["energy"] for w in windows]

plt.figure(figsize=(20,4))  
plt.plot(window_times, window_energies)
plt.show()

We can see clear spikes where the commentator gets louder and more excited!

Now let‘s find the windows with the highest energy above some threshold. We‘ll take a threshold based on the 95th percentile energy:

energy_threshold = np.percentile(window_energies, 95)
highlight_windows = [w for w in windows if w["energy"] >= energy_threshold]

Finally, we‘ll extract the video clips corresponding to the selected windows and combine them into the final highlights reel:

highlight_clips = []
for window in highlight_windows:
    highlight_clip = video.subclip(window["start"], window["end"])
    highlight_clips.append(highlight_clip)

final_highlights = concatenate_videoclips(highlight_clips)
final_highlights.write_videofile("match_highlights.mp4")

That‘s it! We‘ve now generated a highlights video by detecting the most exciting moments using the match commentary audio. The final video will contain all the clips where the commentator‘s voice rose above the threshold level.

Results and Discussion

I tested this approach on soccer and basketball matches from YouTube. After some trial and error to tune the energy threshold, it worked remarkably well! Most of the extracted highlights contained goals, fouls, important shots and other key moments.

The speech analysis approach has several benefits:

• Simple to implement, with minimal lines of code
• Runs quickly, processing matches in faster than real-time
• Doesn‘t require ML so needs no training data or complex models
• Works for any sport with passionate commentary (soccer, basketball, racing, esports)
• Generalizes well to different matches without re-tuning

There are also some limitations to keep in mind:

• Assumes key moments are correlated with exciting commentary (not always true)
• Can miss quieter highlights or include unimportant loud moments
• Struggles with sports lacking continuous energetic commentary (tennis, golf)
• Video quality depends on well-tuned thresholds

With further experimentation, the results could likely be improved by adjusting the window size, overlap and thresholding approach. We could also augment it with video analysis like player or ball tracking.

Overall, speech analysis is a great way to get started with highlight generation in Python, without needing to be a machine learning expert! Give it a try on your favorite sports videos and see what you can extract.

Conclusion and Extensions

In this article, you learned how to automatically generate highlights from sports videos using speech analysis in Python. We covered the basics of audio signals, the short-time energy approach to detecting exciting moments, and implementing it in code with MoviePy.

The speech-based method is simple, fast and effective for sports like soccer, basketball and racing. With further refinement it could be a valuable tool for quickly catching up on matches or generating social media-friendly clips.

There are many potential extensions to experiment with:

• Improving the thresholding method to adapt to different sports and commentators
• Combining with computer vision techniques to confirm extracted highlights
• Applying speech emotion recognition to detect the type of highlight (goal, shot, foul)
• Generating a summary text description to go along with the highlight reel
• Building a real-time highlight extraction tool for live matches

I encourage you to try out the code and ideas from this article! Feel free to apply them to your favorite sports, and share your results and modifications.

What other applications of highlight generation and speech analysis can you think of? Let me know in the comments below!

Originally published March 2024.