Building a Real-time Short News App using HuggingFace Transformers and Streamlit

Introduction

In today‘s fast-paced world, people often don‘t have the time or patience to read through lengthy news articles to stay informed about the latest happenings. We want to quickly skim the key points and move on with our busy lives. Thankfully, advances in natural language processing and AI have made it possible to automatically generate concise summaries of long articles, enabling a more efficient news reading experience.

In this tutorial, we‘ll leverage state-of-the-art AI tools to build ShortNews – an interactive web app that fetches the latest news articles based on a user‘s search query and provides brief auto-generated summaries of each article. ShortNews will make it dead simple to quickly catch up on the latest news without having to read full articles. We‘ll use the HuggingFace Transformers library for text summarization, the Streamlit framework for creating the web UI, the newspaper3k library for article extraction, and the Newscatcher API for retrieving news articles.

By the end of this tutorial, you‘ll have a fully functional short news app that you can run locally and deploy on the cloud for anyone to use. Let‘s dive in!

App Overview

Before we get to the code, let‘s walk through a high-level overview of how ShortNews will work:

1. The user enters a search query (e.g. "Ukraine Russia war") on the app‘s home page
2. The app sends a request to the Newscatcher API to retrieve the top news articles for that query
3. For each retrieved article:

• The article URL is passed to newspaper3k to extract the full article text
• The extracted text is passed through a pre-trained HuggingFace summarization model to generate a 2-3 sentence summary
• The article title, summary, and a button to expand the full article text is displayed on the page

4. The user can read through the summaries, and expand to see the full text of any articles they want to read in more detail

With this flow in mind, let‘s get our development environment set up with the required tools and libraries.

Environment Setup

We‘ll be using Python 3 for this project. If you don‘t have Python installed, you can download it from the official website: https://www.python.org/downloads/

Next, create a new project directory and cd into it:

mkdir shortnews
cd shortnews

Create a new virtual environment and activate it:

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate` instead

Install the required libraries:

pip install streamlit transformers newspaper3k requests

We now have our environment ready with all the tools we need. The only thing missing is an API key to access the Newscatcher API.

Obtaining a Newscatcher API Key

The Newscatcher API provides a free plan that allows unauthenticated access to retrieve news articles. However, I recommend signing up for a free API key to avoid potential throttling.

Go to https://newscatcherapi.com/ and click the "Get API Key" button in the top-right. Create a new account (or log in if you already have one). Once logged in, you should see your API key displayed on the dashboard. Make a note of this key as we‘ll need it later in the code.

Coding the App

Create a new file called app.py in your project directory. This file will contain the entire code for our ShortNews app. Open it in your favorite code editor and let‘s start coding!

First, we‘ll import the required libraries:

import streamlit as st
from transformers import pipeline
import requests
from newspaper import Article

Next, set up the basic Streamlit app:

st.set_page_config(page_title="ShortNews", page_icon="🗞️", layout="wide")
st.title("📰 ShortNews")
st.markdown("Get concise summaries of the latest news articles on any topic!")

Create a sidebar where the user can enter their search query:

query = st.sidebar.text_input("🔍 Search for a topic", "")
num_articles = st.sidebar.slider("Number of articles", min_value=1, max_value=10, value=5, step=1)

Initialize the HuggingFace summarization pipeline:

summarizer = pipeline("summarization", model="facebook/bart-large-cnn")

We‘ll use Facebook‘s pre-trained BART model fine-tuned on the CNN/DailyMail dataset for summarization. This model achieves state-of-the-art results and is freely available on the HuggingFace Model Hub.

Next, we‘ll define a helper function to fetch articles from the Newscatcher API:

def fetch_articles(query, num_articles):
    url = "https://free-news.p.rapidapi.com/v1/search"
    headers = {
        "X-RapidAPI-Key": "YOUR_API_KEY",  # Replace with your Newscatcher API key
        "X-RapidAPI-Host": "free-news.p.rapidapi.com"
    }
    params = {
        "q": query,
        "lang": "en",
        "page": 1,
        "page_size": num_articles
    }
    response = requests.get(url, headers=headers, params=params)
    return response.json()["articles"]

Replace "YOUR_API_KEY" with the API key you obtained earlier.

Now we can fetch articles based on the user‘s search query, extract the text, and generate summaries:

if query:
    with st.spinner(f"Fetching news articles for ‘{query}‘..."):
        articles = fetch_articles(query, num_articles)

    st.subheader(f"Top {num_articles} articles for ‘{query}‘")
    for article_data in articles:
        article = Article(article_data["link"])
        article.download()
        article.parse()

        st.subheader(f"📌 {article.title}")
        summary = summarizer(article.text, max_length=150, min_length=30, do_sample=False)[0]["summary_text"]
        st.write(summary)

        with st.expander("See full article"):
            st.write(article.text)
        st.markdown("---")

This code fetches num_articles articles for the given search query, extracts the title and text using newspaper3k, generates a summary using the HuggingFace summarization pipeline, and displays the title, summary, and expandable full text for each article using Streamlit components.

That‘s it! Our ShortNews app is now fully functional. Let‘s see it in action.

Running the App

In your terminal, run the following command to start the Streamlit app:

streamlit run app.py

This should automatically open a new tab in your browser with the app running. Try searching for different topics and see the summarized articles appear on the page. Pretty nifty, right?

Deploying the App

Lastly, let‘s deploy our app on Streamlit‘s sharing platform so anyone can use it.

If you haven‘t already, create a new GitHub repo for this project and push your code to it.

Then, go to https://share.streamlit.io/ and sign in with your GitHub account.

Click "New app" and select the repo you just created. Choose app.py as the file path and main as the branch. Give your app a catchy name and click "Deploy".

Streamlit will now build and deploy your app. Once it‘s finished, you‘ll see a URL where your app is live. Share this URL with your friends and family so they can try out ShortNews!

The complete code for this app is available on GitHub: https://github.com/yourusername/shortnews

Conclusion

In this tutorial, we built ShortNews – a web app that fetches the latest news articles on any topic and provides auto-generated summaries using AI. We leveraged the HuggingFace Transformers library for state-of-the-art text summarization, the Streamlit framework for quickly building interactive UIs, and the Newscatcher API as our news source.

The complete flow of our app looks like this:

1. User enters a search query
2. App fetches relevant news articles from Newscatcher API
3. newspaper3k extracts the full text of each article
4. HuggingFace summarization pipeline generates a concise summary
5. Article titles, summaries, and expandable full text are displayed in the Streamlit UI

We ran the app locally and then deployed it on Streamlit‘s sharing platform for public access.

There are many potential improvements and extensions to ShortNews, such as:

• Allowing the user to choose the length of summaries
• Adding support for multiple languages
• Providing personalized news recommendations based on user‘s reading history
• Enabling search filters by date, source, category, etc.

Feel free to experiment with these ideas and take ShortNews to the next level!

You can try out the deployed app here: https://share.streamlit.io/yourusername/shortnews/main/app.py

I hope this tutorial gave you a taste of how you can combine AI and web technologies to build powerful apps. If you have any questions or feedback, let me know in the comments below. Happy coding!