Sentiment Analysis with LSTMs: A Deep Dive

Sentiment analysis has become an essential tool in the modern business intelligence stack. With the proliferation of user-generated text data on social media, review sites, support channels and more, organizations can now mine rich attitudinal insights to drive decision making in product development, brand management, customer service, financial modeling, and other key areas.

The field has progressed rapidly in recent years thanks to advancements in deep learning and the availability of large annotated datasets. In particular, Recurrent Neural Networks (RNNs), and specifically Long Short-Term Memory (LSTM) networks, have become the workhorse for sentiment analysis, setting new state-of-the-art benchmarks on academic and industry datasets.

In this post, we‘ll take a comprehensive look at the nuts and bolts of how LSTMs work, walk through a complete code example of training an LSTM for sentiment analysis, share some practical tips and considerations, and discuss real-world applications and case studies.

The Rise of Sentiment Data

First, let‘s look at some statistics that underscore the incredible growth of textual data that businesses are dealing with:

• There are now over 3 billion social media users globally, who collectively post over 500 million tweets, 4 petabytes of Facebook data, and 95 million Instagram photos per day (Source: Domo)
• Over 1 billion online reviews and opinions are posted each year across major sites like Amazon, Yelp, TripAdvisor, and Google (Source: Review Trackers)
• 90% of the world‘s data has been generated in the last two years alone (Source: Forbes)

This deluge of unstructured data presents a huge opportunity for businesses to listen to the voice of their customers, employees, shareholders, and other stakeholders at an unprecedented scale and granularity. Sentiment analysis provides the key to unlock these insights computationally.

Measuring Progress in Sentiment Analysis

The accuracy of sentiment models has improved dramatically thanks to the advent of deep learning. On the binary version of the popular Stanford Sentiment Treebank (SST-2) benchmark, models have reached over 97% accuracy, up from low 80s in the pre-deep learning era.

Here are some highlights of recent state-of-the-art results:

While these numbers are impressive, it‘s important to note that industry applications often deal with more complex, noisy, and domain-specific language than carefully curated academic benchmarks. There is still much room for improvement in practical deployments.

How LSTMs Work

At the core of an LSTM is a memory cell that maintains its state over time, and non-linear gating units that regulate the information flow into and out of the cell. This allows LSTMs to selectively remember and forget context over long sequences.

Mathematically, the LSTM updates for a step at time t given inputs xt, previous hidden state h(t-1), and previous cell state c_(t-1) are as follows:

it = \sigma(W{ii}xt + b{ii} + W{hi}h{t-1} + b_{hi})
ft = \sigma(W{if}xt + b{if} + W{hf}h{t-1} + b_{hf})
ot = \sigma(W{io}xt + b{io} + W{ho}h{t-1} + b_{ho})

\tilde{c}t = \text{tanh}(W{ig}xt + b{ig} + W{hg}h{t-1} + b_{hg})

c_t = ft * c{t-1} + i_t \tilde{c}_t
h_t = o_t \text{tanh}(c_t)

where i_t, f_t, o_t represent the input, forget, and output gates; c_t is the cell state; h_t is the hidden state; and W and b are learned weight and bias parameters. The * operator denotes element-wise multiplication.

Intuitively, the forget gate controls how much of the previous cell state to retain, the input gate controls how much new information to add from the current input and previous hidden state, and the output gate controls how much of the cell state to expose to the next layer or time step. This selective memory allows LSTMs to maintain context over longer distances than vanilla RNNs.

Sentiment Analysis with LSTMs

Now let‘s see how to implement sentiment analysis using an LSTM in TensorFlow 2 / Keras. We‘ll use the canonical IMDB movie review dataset which contains 50k reviews labeled as positive or negative.

The general steps in the modeling pipeline are:

1. Load and preprocess data
2. Tokenize text and convert to sequences
3. Pad sequences to fixed length
4. Define LSTM model architecture
5. Train model
6. Evaluate on test set
7. Inference on new examples

Here are the key snippets of code for each step:

import tensorflow as tf
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing import sequence

# Load data
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=10000)

# Convert to sequences
x_train = sequence.pad_sequences(x_train, maxlen=500)
x_test = sequence.pad_sequences(x_test, maxlen=500)

# Define model
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(10000, 128),
    tf.keras.layers.LSTM(128, dropout=0.2, recurrent_dropout=0.2), 
    tf.keras.layers.Dense(1, activation=‘sigmoid‘)
])

model.compile(loss=‘binary_crossentropy‘,
              optimizer=‘adam‘,
              metrics=[‘accuracy‘])

# Train
model.fit(x_train, y_train,
          batch_size=32,
          epochs=5,
          validation_data=(x_test, y_test))

# Evaluate          
score, acc = model.evaluate(x_test, y_test)

# Inference
review = "This movie was amazing! Highly recommend it."
seq = tokenizer.texts_to_sequences([review])
padded = tf.keras.preprocessing.sequence.pad_sequences(seq, maxlen=500)
pred = model.predict(padded)[0]
print(f‘Review: {review}‘)
print(f‘Sentiment: {"positive" if pred > 0.5 else "negative"} ({pred[0]:.2f})‘)

This simple model achieves around 87% test accuracy, demonstrating the power of LSTMs to capture semantic context.

Practical Considerations

While LSTMs are a go-to architecture for sentiment analysis, there are several practical considerations to keep in mind:

• LSTMs are computationally expensive, especially for long sequences. Techniques like sequence bucketing, model distillation, quantization can help reduce runtime latency.
• Vanilla LSTMs can still struggle with very long-range dependencies. Enhancements like attention, dilated convolutions, and segmented RNNs can help.
• Bidirectional LSTMs that read sequences forwards and backwards can provide additional context, often improving performance.
• Proper hyperparameter tuning (e.g. number of layers, hidden units, learning rate, dropout) is crucial to optimize model quality.
• Transfer learning by fine-tuning pre-trained language models like BERT can significantly boost accuracy, but introduces more complexity.
• Models can easily overfit to specific vocabularies and domains. Continuous monitoring and retraining is important to adapt to data drift.
• Explainability techniques like attention visualization and probing classifiers can help understand model biases and failure modes.

By paying attention to these issues, data scientists can develop more robust, scalable, and maintainable sentiment analysis models.

Use Cases and Case Studies

Sentiment analysis has found wide applicability across industries. Here are a few examples:

• Finance: Hedge funds and banks use sentiment signals from news, analyst reports, earnings calls, and social media to inform trading strategies and risk models. For instance, Accrete.AI provides real-time sentiment analysis of public companies by parsing over 300 million web pages and 60,000 sources daily.

• E-commerce: Online retailers analyze product reviews and customer feedback to identify issues like defects, shipping problems, service gaps, and more. Wise Athena offers an AI platform that aggregates reviews from multiple sites and automatically surfaces actionable insights.

• Healthcare: Doctors and researchers mine sentiment from electronic health records, medical literature, and online forums to gauge treatment effectiveness, side effects, and patient quality of life. Pfizer recently piloted a wearable device that uses sentiment analysis of voice recordings to track disease progression in Parkinson‘s patients.

• Politics: Campaign strategists, pollsters, and civic organizations increasingly use sentiment analysis to measure public opinion on candidates and issues. For example, the MIT Election Lab tracked Twitter sentiment for major presidential candidates in the 2020 election cycle.

As the volume and variety of sentiment-bearing data continues to explode, we can expect to see even more innovative and impactful applications of this technology.

Conclusion

Sentiment analysis is a powerful AI capability that is transforming how organizations sense and respond to the opinions of their stakeholders. LSTMs have emerged as the architecture of choice for this task due to their ability to capture semantic and emotional context.

However, building production-grade sentiment systems requires more than just importing an LSTM layer. Data quality, computational performance, model generalization, and governance are all key challenges.

By understanding the mathematical foundations of LSTMs, benchmarking modeling approaches, and following MLOps best practices, data scientists can harness the full potential of sentiment analysis to deliver highly scalable, accurate and actionable intelligence for their organizations. The field is progressing rapidly and the opportunities for data-driven leaders are immense.