Sentiment Analysis on Tweets with LSTM: A Beginner‘s Guide

Introduction

In the age of social media, millions of people share their thoughts, opinions and feelings online every day. Twitter has become a particularly popular platform for this, with over 500 million tweets sent daily as of 2022. This presents a massive opportunity to gain insights into public sentiment about various topics, from products and services to social issues and political events.

Sentiment analysis, a subfield of natural language processing (NLP), aims to automatically determine the sentiment expressed in a piece of text, whether positive, negative or neutral. By applying sentiment analysis to tweets, we can get a pulse on how people feel about a particular topic at scale.

In this tutorial, we‘ll walk through the process of performing sentiment analysis on tweets using long short-term memory (LSTM) networks, a type of recurrent neural network well-suited for sequence data like text. We‘ll be using the Sentiment140 dataset, which consists of 1.6 million tweets labeled as either positive or negative.

While there are many different approaches to sentiment analysis, from rule-based systems to supervised machine learning models, deep learning techniques like LSTMs have achieved state-of-the-art results in recent years.

By the end of this guide, you‘ll have a solid understanding of the key concepts involved and be able to train your own LSTM model for sentiment analysis. Let‘s get started!

The Sentiment140 Dataset

The Sentiment140 dataset, compiled by Stanford University researchers, contains 1,600,000 tweets extracted using the Twitter API. The tweets were collected by searching for specific emoticons which serve as pseudo-labels:

• 🙂 indicates a positive tweet
• 🙁 indicates a negative tweet

The dataset is split into 1.6M training samples and 359 test samples. Columns in the dataset include:

• target: the sentiment label (0 = negative, 4 = positive)
• ids: unique tweet identifier
• date: date tweet was posted
• flag: format of tweet (can ignore)
• user: username who posted tweet
• text: text of the tweet

For our purposes, we only need the target (label) and text columns. The dataset is relatively balanced between positive and negative tweets.

Data Preprocessing

Cleaning and preprocessing the raw tweet text is a crucial step before feeding it into our model. Some key preprocessing steps include:

Text cleaning

Tweets are notoriously noisy, containing URLs, HTML tags, user mentions (@), hashtags (#), and other metadata irrelevant to sentiment. We can use regular expressions to strip these out and convert the tweet to lowercase:

def clean_text(text):
    text = re.sub(r‘http\S+‘, ‘‘, text) # remove URLs
    text = re.sub(r‘@\w+‘, ‘‘, text) # remove mentions  
    text = text.lower() # convert to lowercase
    text = re.sub(r‘[^a-zA-Z0-9\s]‘, ‘‘, text)  # only keep alphanumeric chars and spaces
    return text

Removing stop words

Stop words are common words that appear frequently in text but don‘t contribute much to its meaning, like "the", "and", "is". We can remove these using NLTK‘s built-in list:

from nltk.corpus import stopwords

stop_words = stopwords.words(‘english‘) 

def remove_stopwords(text):
    return " ".join([word for word in text.split() if word not in stop_words])

Stemming and lemmatization

Stemming and lemmatization both aim to reduce words to their base or dictionary form (e.g. "running" to "run"). This helps normalize the text and reduce the vocabulary size. While stemming just chops off affixes, often producing incomplete words, lemmatization uses detailed dictionary information to return complete words:

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

def lemmatize_text(text):
    return " ".join([lemmatizer.lemmatize(word) for word in text.split()])  

Tokenization and padding

Before we can feed the tweets into our LSTM model, we need to tokenize them by converting each word to an integer ID. We can use Keras‘ Tokenizer class for this:

from tensorflow.keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer()
tokenizer.fit_on_texts(cleaned_tweets)

tweet_sequences = tokenizer.texts_to_sequences(cleaned_tweets) 

We also need to pad or truncate each sequence to a fixed length, since the input to the LSTM must be a rectangular matrix:

from tensorflow.keras.preprocessing.sequence import pad_sequences

max_length = 50 
padded_tweets = pad_sequences(tweet_sequences, 
                              maxlen=max_length,
                              padding=‘post‘,
                              truncating=‘post‘)

Word Embeddings

While we could one-hot encode each word, this leads to massive sparse vectors that don‘t capture semantic similarity between words. Instead, we can use pre-trained word embeddings like GloVe or Word2Vec. These embeddings map each word to a dense vector, such that semantically similar words have similar vectors.

We can load the pre-trained GloVe embeddings and create an embedding matrix mapping words in our vocabulary to their GloVe vectors:

glove_dir = ‘/path/to/glove.twitter.27B‘

embeddings_index = {}
f = open(os.path.join(glove_dir, ‘glove.twitter.27B.200d.txt‘))
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype=‘float32‘)
    embeddings_index[word] = coefs
f.close()

vocab_size = len(tokenizer.word_index) + 1
embedding_dim = 200

embedding_matrix = np.zeros((vocab_size, embedding_dim))
for word, i in tokenizer.word_index.items():
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        embedding_matrix[i] = embedding_vector

Building the LSTM Model

Now that we have our data preprocessed and word embeddings ready, we can build our LSTM model. LSTMs are a type of recurrent neural network capable of learning long-term dependencies in sequential data.

The key component of LSTMs is the memory cell, which can store information over long periods of time. The cell is regulated by three gates:

• Forget gate: decides what information to discard from the cell
• Input gate: decides what new information to store in the cell
• Output gate: decides what information to output from the cell

This gating mechanism allows LSTMs to selectively remember or forget information over many time steps, making them well-suited for tasks like sentiment analysis where capturing long-range context is important.

Here‘s how we can define a simple LSTM model in Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Embedding

model = Sequential()
model.add(Embedding(vocab_size, 
                    embedding_dim, 
                    input_length=max_length, 
                    weights=[embedding_matrix],
                    trainable=False))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))  
model.add(Dense(1, activation=‘sigmoid‘))

model.compile(loss=‘binary_crossentropy‘,
              optimizer=‘adam‘,
              metrics=[‘accuracy‘])

This model first passes the input sequences through an Embedding layer initialized with our pre-trained GloVe vectors. The embedded sequences are then passed through an LSTM layer with 128 units, using dropout for regularization. Finally, the LSTM outputs are passed through a Dense output layer with sigmoid activation to produce the sentiment prediction.

Since this is a binary classification task, we use binary cross-entropy loss and Adam optimization.

Training and Evaluation

We can now train our model on the preprocessed and labeled tweet data:

history = model.fit(padded_tweets, labels, 
                    epochs=5,
                    batch_size=32,
                    validation_split=0.1)

After training for 5 epochs, our model achieves around 78% validation accuracy. Not bad for a simple model! Of course, there are many ways we could improve this, such as:

• Using a more powerful pre-trained model like BERT as the base
• Stacking additional recurrent layers to capture higher-level features
• Using attention mechanisms to focus on the most relevant words
• Fine-tuning the model on a domain-specific dataset

To get a more complete picture of our model‘s performance, we can look at a confusion matrix:

from sklearn.metrics import confusion_matrix

preds = (model.predict(test_tweets) > 0.5).astype("int32")
cm = confusion_matrix(test_labels, preds)

sns.heatmap(cm, annot=True, fmt=‘d‘, cmap=‘Blues‘, square=True)

We can see that the model is slightly better at predicting positive sentiment than negative, but overall does a decent job on both classes.

Making Predictions on New Tweets

To predict the sentiment of a new incoming tweet, we can simply pass it through the same preprocessing pipeline and feed it into our trained model:

def predict_sentiment(tweet):
    cleaned = clean_text(tweet)
    stemmed = stem_text(cleaned)
    padded = pad_sequences(tokenizer.texts_to_sequences([stemmed]), maxlen=max_length)

    pred = model.predict(padded)[0][0]
    return "Positive" if pred > 0.5 else "Negative"  

We could even expose this as a REST API endpoint using a web framework like Flask:

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route(‘/predict‘, methods=[‘POST‘])
def predict():
    tweet = request.json[‘tweet‘]
    sentiment = predict_sentiment(tweet)
    return jsonify({"sentiment": sentiment})

A user could then hit this endpoint with a raw tweet and get back its predicted sentiment in real-time!

Conclusion and Next Steps

In this guide, we‘ve seen how to train an LSTM model for sentiment analysis on tweets in a few simple steps:

1. Preprocess the raw tweet text by cleaning, removing stop words, stemming, and tokenizing
2. Load pre-trained word embeddings to represent tweets as dense vectors
3. Feed tweet vectors into an LSTM model and train on labeled data
4. Evaluate model performance on holdout test data
5. Use trained model to predict sentiment of new tweets

Of course, this just scratches the surface of what‘s possible with modern NLP techniques. Some potential next steps to explore:

• Experiment with more advanced model architectures like Transformers
• Leverage transfer learning by fine-tuning pre-trained language models
• Expand to multi-class sentiment (positive, neutral, negative) or emotion analysis
• Deploy model as a real-time sentiment analysis application

With the vast amount of opinionated text data available on social media and beyond, sentiment analysis will only become more valuable for businesses and organizations looking to understand public perception. Hopefully this guide provides a solid foundation to begin applying it yourself!