Emotion Detection using Bidirectional LSTM and Word2Vec
Emotion detection, also known as sentiment analysis, is the process of using natural language processing and machine learning techniques to automatically identify and categorize the emotional sentiment expressed in a piece of text. Emotion detection has become an increasingly important task, with applications ranging from social media monitoring and customer service to mental health tracking and affective computing.
There are various approaches to building emotion detection models, but one particularly powerful method is to use a bidirectional long short-term memory (LSTM) neural network in combination with word embeddings like Word2Vec or GloVe. In this article, we‘ll provide a comprehensive guide on how to build an emotion detection model using bidirectional LSTMs and the GloVe word embeddings, walking through each step of the process in detail.
Understanding Bidirectional LSTMs
To begin, let‘s first take a closer look at what bidirectional LSTMs are and how they work. LSTMs are a type of recurrent neural network (RNN) architecture that are well-suited for modeling sequential data, such as time series or natural language. They work by maintaining an internal "memory state" that allows them to learn long-term dependencies in the input sequence.
Standard LSTMs process the input sequence in one direction, typically from left to right. Bidirectional LSTMs (Bi-LSTMs), on the other hand, process the sequence in both directions simultaneously. This allows the model to have both backward and forward information about the sequence at every time step.
Concretely, a Bi-LSTM consists of two LSTMs – one that processes the input sequence forwards and one that processes it backwards. The outputs of the two LSTMs are then typically concatenated together at each time step. This allows the hidden state at each time step to capture both past and future context, often leading to better performance on sequence modeling tasks compared to unidirectional LSTMs.
Word Embeddings with Word2Vec and GloVe
In order to feed textual data to our Bi-LSTM model, we first need to convert the raw text into a numeric representation. While we could use a simple one-hot encoding, where each word in the vocabulary is represented as a binary vector, this is often ineffective for NLP tasks. One-hot encodings result in extremely sparse, high-dimensional vectors that fail to capture the semantic relationships and similarities between words.
Word embeddings offer a much more effective way to represent words as dense, low-dimensional vectors that encode semantic meaning. Two of the most popular word embedding algorithms are Word2Vec and GloVe.
Word2Vec, developed by researchers at Google, uses a shallow neural network to learn word embeddings in an unsupervised manner from a large corpus of text. The key idea is to predict the context words given a center word, or vice versa. This allows semantically similar words to end up with similar vector representations.
GloVe, which stands for "Global Vectors for Word Representation", is another prominent word embedding technique. Developed by researchers at Stanford, GloVe works by factorizing a matrix of word-word co-occurrence statistics. The resulting word vectors capture both local context information, like Word2Vec, as well as global statistics across the full corpus.
In practice, pre-trained word embeddings like those trained on enormous corpora like Wikipedia or Common Crawl are often used. This allows transfer learning and leverages the semantic information learned from vast amounts of unlabeled text. In this article, we‘ll be using the pre-trained GloVe embeddings, specifically the glove.6B.100d version which contains 400,000 words represented as 100-dimensional vectors trained on 6 billion tokens.
Building the Emotion Detection Model
With that background in place, we‘re now ready to step through the process of building the emotion detection model. We‘ll be using the Python deep learning library Keras with a TensorFlow backend. The full code is available on GitHub, but we‘ll walk through the key steps here.
1. Loading and Preprocessing the Data
The first step is to load our emotion-labeled text dataset and apply any necessary preprocessing. A common dataset used for emotion detection is the "Emotions Dataset for NLP" from Kaggle, which contains 20,000 English Twitter messages labeled with 6 emotions: anger, fear, joy, love, sadness, and surprise.
We‘ll load this dataset into a Pandas DataFrame and apply some basic preprocessing steps, such as:
- Converting all text to lowercase
- Removing punctuation, special characters, URLs, etc.
- Converting emojis/emoticons to text (e.g. 🙂 → happy)
- Expanding contractions (e.g. I‘m → I am)
- Lemmatizing words (e.g. running → run)
Here‘s an example of applying these preprocessing steps using Python:
import re
import pandas as pd
from text_hammer import TextHammer
df = pd.read_csv(‘data/emotions_dataset.csv‘)
th = TextHammer()
def preprocess(text):
text = str(text).lower()
text = th.remove_emails(text)
text = th.remove_html_tags(text)
text = th.remove_special_chars(text)
text = th.cont_exp(text)
text = th.make_base(text)
return text
df[‘text‘] = df[‘text‘].apply(preprocess) 2. Tokenization and Sequencing
Next, we need to tokenize the preprocessed text data and convert the tokens to integer sequences. Tokenization refers to splitting the text up into individual units, which could be words, characters, or subwords. The simplest approach is to split on whitespace to get word-level tokens.
We‘ll use the Keras Tokenizer class to build the mapping from words to integers. We set a maximum vocabulary size of 10,000, meaning we‘ll only consider the 10,000 most common words in the corpus.
from tensorflow.keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(num_words=10000, oov_token=‘UNK‘)
tokenizer.fit_on_texts(df[‘text‘])
sequences = tokenizer.texts_to_sequences(df[‘text‘])The texts_to_sequences function converts each text into a sequence of integers. We can also use the tokenizer.word_index property to get the mapping of words to their integer tokens.
3. Padding Sequences
The sequences need to be padded to a uniform length to be fed into the Bi-LSTM model. We‘ll pad the sequences to a maximum length of 300 tokens. Sequences longer than this will be truncated and sequences shorter will be padded with zeros at the end.
from tensorflow.keras.preprocessing.sequence import pad_sequences
max_length = 300
padded_sequences = pad_sequences(sequences, maxlen=max_length,
padding=‘post‘, truncating=‘post‘)4. Loading Pre-trained GloVe Embeddings
Rather than learning the word embeddings from scratch, we‘ll load the pre-trained GloVe embeddings. We‘ll build a embedding matrix where each row corresponds to a word in our vocabulary and the columns are the embedding dimensions.
Words in our vocabulary that are also in the GloVe embeddings will have their embedding vector loaded into the matrix. For out-of-vocabulary words, we‘ll just use a vector of zeros.
import numpy as np
embedding_dim = 100
vocabulary_size = min(len(tokenizer.word_index)+1, 10000)
embedding_matrix = np.zeros((vocabulary_size, embedding_dim))
for word, i in tokenizer.word_index.items():
if i < vocabulary_size:
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
embedding_matrix[i] = embedding_vector5. Building the Bi-LSTM Model
We‘re now ready to build the Bi-LSTM model using the Keras Sequential API. The key steps are:
-
Add an Embedding layer: This layer maps the sequence of integer tokens to a sequence of dense embedding vectors using the pre-trained GloVe embedding matrix. Set
trainable=Falseto keep the embeddings fixed. -
Add a Bidirectional LSTM layer: This applies the LSTM in both directions across the sequence. We‘ll use 200 hidden units in each direction.
-
Add a Dropout layer for regularization.
-
Add a Dense output layer with softmax activation to get the prediction probabilities for each emotion class.
Here‘s the code to build this model architecture:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM
from tensorflow.keras.layers import Dense, Dropout
model = Sequential()
model.add(Embedding(vocabulary_size, embedding_dim,
weights=[embedding_matrix],
trainable=False))
model.add(Bidirectional(LSTM(200, return_sequences=False)))
model.add(Dropout(0.25))
model.add(Dense(6, activation=‘softmax‘))
model.compile(loss=‘categorical_crossentropy‘,
optimizer=‘adam‘, metrics=[‘accuracy‘])6. Training the Model
To train the model, we first need to one-hot encode the emotion labels. We then split the data into training and validation sets and train the model for some number of epochs.
We‘ll use an EarlyStopping callback to stop training when the model stops improving on the validation set. We‘ll also use ModelCheckpoint to save the best model seen during training.
from tensorflow.keras.utils import to_categorical
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
# Convert labels to categorical one-hot encoding
label_map = {‘anger‘: 0, ‘fear‘: 1, ‘joy‘: 2,
‘love‘: 3, ‘sadness‘: 4, ‘surprise‘: 5}
y = to_categorical([label_map[label] for label in df[‘emotion‘]])
# Split the data into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(
padded_sequences, y, test_size=0.2, random_state=1)
callbacks = [
EarlyStopping(patience=5, verbose=1, restore_best_weights=True),
ModelCheckpoint(‘models/lstm_w2v_best.h5‘,
save_best_only=True, verbose=1)
]
history = model.fit(X_train, y_train,
validation_data=(X_val, y_val),
batch_size=128, epochs=50,
callbacks=callbacks, verbose=1)7. Evaluating the Model
Finally, we evaluate the trained model on a separate test set. We‘ll report the overall accuracy as well as a confusion matrix to see which emotions the model struggles with.
from sklearn.metrics import accuracy_score, confusion_matrix
y_pred = model.predict(X_test)
y_pred = np.argmax(y_pred, axis=1)
y_true = np.argmax(y_test, axis=1)
print("Test Accuracy: {:.3f}".format(accuracy_score(y_true, y_pred)))
print("Confusion Matrix:\n", confusion_matrix(y_true, y_pred))Tips for Improving Model Performance
There are a number of ways to potentially improve the performance of this baseline emotion detection model:
Use a larger or more appropriate dataset: The model will only be as good as the data it‘s trained on. Using a larger dataset with more examples of each emotion could help, as well as ensuring the data is representative of the type of text the model will be used for (e.g. Tweets vs. news articles).
Experiment with different model architectures: We used a fairly simple Bi-LSTM architecture here, but there are many other options. Adding more LSTM layers, using a CNN or Transformer-based architecture, or including an attention mechanism are all worth exploring.
Fine-tune the word embeddings: We used pre-trained GloVe embeddings here, but fine-tuning them on the specific emotion dataset (by setting trainable=True in the Embedding layer) may help the model learn more emotion-specific semantic relationships between words.
Experiment with different hyperparameters: Trying different values for things like the embedding dimension, LSTM hidden size, dropout rate, batch size, and learning rate could potentially lead to improved performance. Systematic hyperparameter tuning using a validation set is recommended.
Use an ensemble of models: Training multiple models with different architectures or on different subsets of the data and combining their predictions (e.g. by majority vote) at test time can be a powerful technique.
Advanced Approaches with BERT
While the Bi-LSTM with GloVe approach can work well, the current state-of-the-art in many NLP tasks, including emotion detection, is to use large pre-trained language models like BERT.
BERT (Bidirectional Encoder Representations from Transformers) is a deep bidirectional Transformer model that is pre-trained on a massive amount of unlabeled text data. The pre-training allows BERT to learn a powerful, generalizable representation of language that can then be fine-tuned for specific tasks like emotion detection with relatively little labeled data.
The general approach is to add a classification layer on top of the pre-trained BERT model and fine-tune the entire model end-to-end on the emotion dataset. This often outperforms models trained from scratch and allows leveraging the linguistic knowledge learned by BERT from huge unsupervised corpora.
Using BERT for emotion detection is a more advanced topic that we don‘t have space to fully cover here, but the key steps are:
- Load a pre-trained BERT model (e.g. from HuggingFace Transformers)
- Add a sequence classification head on top of the BERT outputs
- Prepare the data into the required format (tokenize with BERT tokenizer, etc.)
- Fine-tune the model on the emotion dataset
- Evaluate and iterate
There are also newer, even more powerful language models like XLNet, RoBERTa, ALBERT, etc. that could be used in a similar way. Exploring these advanced transfer learning approaches is a promising direction for further improving emotion detection performance.
Conclusion
In this article, we‘ve provided a comprehensive guide on building an emotion detection model using the powerful combination of bidirectional LSTMs and GloVe word embeddings. The key steps are:
- Preprocess the text data
- Tokenize the text and convert to sequences
- Pad the sequences
- Load pre-trained word embeddings
- Build a Bi-LSTM model
- Train the model
- Evaluate on a test set
We‘ve also covered a number of ways the basic model could potentially be improved, as well as discussing more advanced approaches using large pre-trained models like BERT.
Emotion detection is a challenging but important NLP task with many interesting applications. While we‘ve focused on the core techniques here, there are many areas for further exploration, such as using more sophisticated preprocessing techniques, leveraging character or subword embeddings, cross-domain and cross-lingual emotion detection, few-shot learning approaches, and so on. We encourage interested readers to experiment with building their own emotion detection models and push the state-of-the-art in this exciting field.
