Building a Tweet Sentiment Analysis Web App with Streamlit

Introduction

The rise of social media has led to an explosion of publicly available data capturing people‘s thoughts, feelings and opinions on every imaginable topic. This presents a golden opportunity for businesses, researchers, and policymakers to understand public sentiment in real-time and at scale. However, the sheer volume of data means manually analyzing sentiment is impractical, if not impossible.

This is where natural language processing (NLP) and machine learning come in. By training models to automatically classify the sentiment of text, we can efficiently analyze massive datasets and derive actionable insights. And by integrating these models into user-friendly web applications, we can empower people to engage in sentiment analysis without any coding or data science experience required.

In this post, we‘ll walk through a practical example of how to build a sentiment analysis web app using the Streamlit framework and a pre-trained NLP model. We‘ll focus on analyzing sentiment for tweets, but the same approach could be applied to reviews, comments, news articles, or any other text data. Let‘s get started!

What is Sentiment Analysis?

Sentiment analysis, also known as opinion mining or emotion AI, is the use of NLP, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. In simpler terms, it‘s the process of determining whether a piece of text is positive, negative or neutral.

Some common applications of sentiment analysis include:

• Social media monitoring to understand how people feel about a brand, product, or topic
• Customer feedback analysis to identify pain points and improve products and services
• Market research and competitive analysis
• Political campaign monitoring and polling
• Detecting early warning signs of problems like customer churn or employee turnover

The sentiment analysis market is expected to grow from $3.2B in 2020 to $7.1B by 2025, at a CAGR of 17.5% during the forecast period (Source: MarketsandMarkets Research). This growth is driven by the increasing availability of social data, advancements in AI/ML technologies, and growing enterprise demand for real-time customer insights.

While humans are generally good at identifying sentiment in text, we are prone to biases and inconsistencies. We also get fatigued quickly when faced with analyzing sentiment at a large scale. Machine learning models, on the other hand, can be trained to predict sentiment with high accuracy and can scale to process massive volumes of data in real-time.

Introduction to Streamlit

Streamlit is an open-source Python library that makes it easy to build interactive web applications for machine learning and data science. It allows developers to create beautiful apps with simple Python scripts and minimal web development experience.

Some key benefits of using Streamlit for machine learning web apps:

• Simple and intuitive API that makes app development fast
• Supports rich output formats like plots, tables, metrics, and media out-of-the-box
• Enables interactive widgets for accepting user input and controlling app state
• Built-in caching for efficient computation and fast app performance
• Easy deployment to public cloud platforms or behind a firewall

Streamlit has seen rapid adoption in the data science community, with over 10,000 GitHub stars and 200,000 downloads per month as of March 2021. It is used by data scientists at companies like Uber, Twitter, Ford, and Stitch Fix to build internal tools and customer-facing applications.

Project Overview

The goal of this project is to build a Streamlit web app that can predict the sentiment of user-entered tweets as either positive or negative. The app will utilize a pre-trained logistic regression model that has been trained on 1.6 million labeled tweets from Kaggle to make predictions.

The final application will look something like this:

Here‘s a high-level overview of the steps we‘ll follow:

1. Install and import required libraries
2. Load the pre-trained logistic regression model and NLP tools
3. Define a function to preprocess tweet text and make sentiment predictions
4. Build the Streamlit web app UI for accepting user input and displaying results
5. Deploy the app to a public cloud platform

The complete code is available in this GitHub repository.

Building the Streamlit App

Installing and Importing Libraries

First, make sure you have Python 3.6+ and the latest version of pip installed. Then install the required libraries in a virtual environment:

pip install streamlit pandas scikit-learn spacy

Next, import the libraries at the top of your app.py script:

import streamlit as st
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
import pickle
import spacy

Loading the Model

The pre-trained logistic regression model and count vectorizer have been saved as pickle files in the models directory. We can load them using the pickle.load function:

vectorizer = pickle.load(open(‘models/vectorizer.pkl‘, ‘rb‘))
model = pickle.load(open(‘models/model.pkl‘, ‘rb‘))
nlp = spacy.load(‘en_core_web_sm‘)

We‘re using spaCy, a popular NLP library, to load a pre-trained pipeline for processing English text. This will be used to tokenize and lemmatize tweet text before passing to the model.

Making Predictions

Next, we‘ll define a helper function to preprocess raw tweet text and get a sentiment prediction from the loaded model:

def predict_sentiment(tweet):
    tweet = str(tweet).lower().replace(‘#‘, ‘‘).replace(‘@‘, ‘‘)
    doc = nlp(tweet)
    clean_tokens = [token.lemma_ for token in doc if not token.is_stop and not token.is_punct]
    clean_tweet = ‘ ‘.join(clean_tokens)
    encoded_tweet = vectorizer.transform([clean_tweet])
    pred = model.predict(encoded_tweet)[0]
    return ‘Positive‘ if pred == 1 else ‘Negative‘

The predict_sentiment function does the following:

1. Converts tweet to lowercase and removes # and @ symbols
2. Tokenizes the tweet using spaCy and lemmatizes each token
3. Removes stop words and punctuation
4. Joins the cleaned tokens back into a string
5. Encodes the cleaned tweet using the loaded CountVectorizer
6. Makes a prediction using the loaded logistic regression model
7. Returns ‘Positive‘ if prediction is 1, else returns ‘Negative‘

Building the UI

Now we‘re ready to build the Streamlit app UI! Add the following code to app.py:

st.set_page_config(page_title=‘Tweet Sentiment Analysis‘, page_icon=‘:smiley:‘, layout=‘wide‘)

st.title(‘Tweet Sentiment Analysis‘)
st.markdown(‘Enter tweet text below to get sentiment prediction.‘)

with st.form(key=‘my_form‘):
    text_input = st.text_area(label=‘Enter tweet text:‘, height=200)
    submit_button = st.form_submit_button(label=‘Get Sentiment‘)

if submit_button:    
    sentiment = predict_sentiment(text_input)
    st.markdown(f‘Predicted sentiment: **{sentiment}**‘)
    st.markdown(‘Probability of being positive:‘)
    proba = model.predict_proba(vectorizer.transform([text_input]))[0]
    pos_proba = proba[1] 
    st.progress(pos_proba)

Let‘s break this down:

• st.set_page_config sets the page title, favicon, and layout
• st.title and st.markdown display the app title and instructions
• st.form creates a form that includes a text area input and a submit button
• When the form is submitted, we call our predict_sentiment function and display the predicted sentiment
• We also display the predicted probability of the tweet being positive using st.progress

To run the app, simply execute streamlit run app.py from the terminal. This will open the app in a new browser tab.

Deploying the App

To make the app publicly accessible, we need to deploy it to a cloud hosting provider. Streamlit integrates seamlessly with platforms like Heroku, AWS, GCP, and Azure.

For this example, we‘ll deploy to Heroku, which offers a generous free tier. Here are the steps:

1. Sign up for a free Heroku account and install the Heroku CLI
2. Run heroku login and enter your credentials
3. Navigate to your app directory and create a git repo:

   git init
   heroku git:remote -a your-app-name

4. Create a requirements.txt file with the required Python libraries
5. Create a Procfile with the command to run the app:

   web: sh setup.sh && streamlit run app.py

6. Create a setup.sh file to set up the spaCy model:

   #!/bin/bash
   python -m spacy download en_core_web_sm

7. Add all files to git and commit changes:

   git add .
   git commit -m "Initial commit" 

8. Push to Heroku:

   git push heroku master

That‘s it! Your app is now live at https://your-app-name.herokuapp.com. See the Streamlit deployment guide for more details and options.

Improving the Model

The logistic regression model used in this example was trained on a dataset of 1.6 million tweets and achieves an accuracy of 79% on a held-out test set. While this is a good baseline, there are many ways we could potentially improve model performance, such as:

• Using a larger and more diverse training dataset
• Experimenting with more advanced NLP techniques like word embeddings, transfer learning, or transformers
• Fine-tuning model hyperparameters using grid search or random search
• Doing additional text preprocessing like handling emojis, speling correction, and hashtag segmentation
• Ensembling multiple models together

To give an example, a recent study was able to achieve 91% accuracy on the same dataset by using a pre-trained BERT model fine-tuned on the sentiment labels (Source: Twitter Sentiment Analysis Using a Modified BERT Model).

It‘s also important to continuously monitor model performance on new data and retrain periodically on new examples to combat model drift. Tools like Evidently can help automate the process of measuring model degradation over time.

Ethical Considerations

When building any AI system, it‘s critical to consider potential ethical implications and unintended consequences. This is especially true with sentiment analysis models, which aim to quantify highly subjective human opinions.

Some ethical risks to watch out for:

• Biased predictions based on gender, race, or other sensitive attributes
• Misuse of sentiment predictions to manipulate public opinion or discriminate against groups
• Lack of transparency in how sentiment labels are defined and models are evaluated
• Invasion of privacy from analyzing personal communications without consent

To mitigate these risks, we should strive to:

• Use diverse and representative training data
• Test models for fairness and robustness to distribution shift
• Provide clear definitions of sentiment labels and document potential failure modes
• Give users control over if and how their data is used
• Continue to monitor model performance and impact after deployment

For a deeper dive into the ethical challenges of sentiment analysis, check out this paper on The Ethics of Sentiment Analysis.

Extensions and Next Steps

There are many ways to extend and improve upon this basic tweet sentiment analysis app, such as:

• Adding support for batch processing of tweets from a CSV file or API
• Generating visualizations and summary statistics of sentiment over time or by topic
• Enabling user authentication and persistence of past analyses
• Adding the ability to compare sentiment across different search terms or handles
• Deploying the model as a scalable REST API using a framework like FastAPI or Flask
• Containerizing the app using Docker for easier deployment and reproducibility

I‘d encourage you to fork the code repo and experiment with adding your own features and improvements!

Conclusion

In this post, we saw how to build an interactive web app for analyzing the sentiment of tweets using the Streamlit framework and a pre-trained NLP model. With just a few dozen lines of Python, we were able to create an app that provides real-time predictions and visualizations with an intuitive user interface.

The complete code is available on GitHub and a live demo can be accessed here.

Tools like Streamlit are democratizing access to machine learning and empowering data scientists to rapidly build and share data-driven applications. By reducing the barrier to entry for ML app development, we can put the power of data science into more people‘s hands and accelerate the positive impact of AI on society.

This is an exciting time to be working at the intersection of data science, ML engineering, and application development. As ML models become more powerful and platforms like Streamlit make it easier to productionize those models, there will be no shortage of opportunities to build innovative AI-powered applications. I hope this post inspires you to go out and build your own!