An Empirical Study of Machine Learning Classifiers for Tweet Sentiment Classification

Sentiment analysis, the task of automatically determining the opinion or emotion expressed in a piece of text, has become a critical tool for organizations looking to gain insights from the vast amount of user-generated content on social media. Twitter, with over 330 million monthly active users generating over 500 million tweets per day, is a particularly rich source of real-time public opinion on everything from products and services to social and political issues.[^1]

However, analyzing sentiment from tweets poses unique challenges compared to traditional text data. Tweets are short (limited to 280 characters), noisy (with slang, misspellings, and hashtags), and often use sarcasm and irony, making it difficult for even human annotators to determine their true sentiment.[^2] Additionally, the sheer volume and velocity of tweet data requires scalable and automated analysis techniques.

In this post, we‘ll explore an empirical study comparing various machine learning approaches for tweet sentiment classification. We‘ll use a large labeled dataset of tweets about U.S. airlines and evaluate performance across different text embedding schemes and classification algorithms. Finally, we‘ll discuss key insights and best practices for building robust social media sentiment analysis systems.

The Twitter US Airline Sentiment Dataset

The Twitter US Airline Sentiment Dataset contains 14,640 tweets about major U.S. airlines (American, Delta, Southwest, United, US Airways, and Virgin America) collected in February 2015. Each tweet was manually labeled as either positive, neutral, or negative in sentiment towards the airline mentioned.

Here are some key statistics about the dataset:

• 62% of tweets are negative, 21% are neutral, and 17% are positive
• Tweets contain an average of 18 words and 120 characters
• The most common words are "flight" (6556 occurrences), "delayed" (3860), and "cancelled" (3676)
• American Airlines has the most negative sentiment (64% negative), while Virgin America has the most positive (28% positive)

This class imbalance, with the majority of tweets being negative, is common in customer service domains like airlines. Customers are more likely to post on social media to complain or vent frustration than to praise good service.[^3]

Text Preprocessing

Before applying any machine learning models, it‘s critical to clean and normalize the raw tweet text. We used the following preprocessing steps:

1. Lowercasing: Convert all text to lowercase to treat words like "Flight" and "flight" the same
2. Username and URL replacement: Replace Twitter usernames like "@united" and URLs with special tokens to avoid treating them as informative words
3. Punctuation and digit removal: Remove all punctuation marks and digits, which are not directly relevant to sentiment. Keep hashtags since they often contain opinionated words.
4. Tokenization: Split tweets into individual words or tokens using NLTK‘s TweetTokenizer
5. Stopword removal: Remove common English words like "the" and "and" which occur frequently but convey little semantic meaning
6. Stemming: Reduce words to their base or dictionary form, so that "delayed", "delay", and "delays" are all treated as the same word. We used the Porter stemmer from NLTK.

Here‘s an example of a raw tweet and its preprocessed form:

Original:

@united Flight UA1559 delayed 2hrs due to mechanical issue. Missed connection in ORD! Horrible customer service! Never flying #United again!

Preprocessed:

user flight ua1559 delay 2hr due mechan issu miss connect airport horribl custom servic never fly #unit again

Converting Tweets to Numeric Vectors

Since machine learning models require numeric input, the preprocessed tweet text needs to be converted into fixed-length numeric vector representations. There are several approaches for creating these embeddings that capture semantic meaning and relationships between words:[^4]

1. Bag-of-Words and TF-IDF

The simplest method is bag-of-words, which represents each document (tweet) as a vector of word counts. For a vocabulary of size $V$ (unique words across all documents), each document becomes a $V$-dimensional sparse vector, with element $i$ being the count of word $w_i$ in that document.

Term Frequency-Inverse Document Frequency (TF-IDF) extends this idea by weighting word counts by their frequency in a document divided by their frequency across all documents:

$\text{tfidf}(w,d,D) = \text{tf}(w,d) \times \text{idf}(w,D)$

where $\text{tf}(w,d)$ is the raw count of word $w$ in document $d$, and $\text{idf}(w,D) = \log \frac{N}{|{d \in D: w \in d}|}$ with $N$ being the total number of documents and $|{d \in D: w \in d}|$ the number of documents containing word $w$.

2. Word2Vec Embeddings

Word2Vec is a neural network model that learns low-dimensional dense vector representations of words.^5 It‘s trained in an unsupervised way to predict a center word given its context words within a sliding window (Skip-gram), or predict the context words given a center word (Continuous Bag-of-Words/CBOW).

After training on a large text corpus, the learned word vectors exhibit semantic relationships, such that similar words have similar vectors. By taking the element-wise average of the word vectors in a document, we can obtain a fixed-length document vector as well.

3. Doc2Vec Embeddings

Doc2Vec is an extension of Word2Vec that jointly learns vector representations of both words and documents.^6 In addition to the word vectors, it trains a separate "paragraph vector" for each document. When training, the paragraph vector is averaged or concatenated with the word vectors to predict the next word, so that the document embedding captures its overall semantic content.

4. Topic Models

Latent Dirichlet Allocation (LDA) is a generative probabilistic model that represents documents as mixtures of $K$ latent topics, and topics as distributions over words.[^7] Each document is associated with a $K$-dimensional vector $\theta_d$ indicating its topic proportions, and each topic is associated with a $V$-dimensional vector $\phi_k$ representing its word probabilities. These vectors are learned by maximizing the likelihood of the observed documents under the model.

For our experiments, we trained 100-dimensional Word2Vec embeddings using the Gensim library on the preprocessed tweet corpus. We used the Skip-gram architecture with a context window size of 5 and negative sampling. We also trained a 100-dimensional Doc2Vec model with a window size of 10 and negative sampling. For LDA, we used the gensim implementation with 25 topics.

Classification Models

With the tweets converted to numeric vectors, we can train machine learning models to predict the sentiment of new incoming tweets. We evaluated a range of classical and deep learning text classification models:

• Logistic Regression: A linear model that learns a weight vector to make a binary prediction by sigmoid thresholding. We used logistic regression with L2 regularization from scikit-learn.

• Support Vector Machines (SVM): Maximal-margin classifiers that learn a hyperplane to separate classes in a high-dimensional space. We tried linear, polynomial, and RBF kernels using scikit-learn‘s SVC class.

• Naive Bayes: Probabilistic classifiers that apply Bayes‘ theorem with strong independence assumptions between features. We used Gaussian and Multinomial variants from scikit-learn.

• Random Forests: An ensemble of decision trees trained on random subsets of data. Random forests are robust to overfitting and nonlinearities. We used scikit-learn‘s RandomForestClassifier with 100 trees.

• Gradient Boosting: Another tree-based ensemble method that iteratively fits decision trees to minimize a loss function. We used XGBoost with 100 boosting rounds and a max depth of 6.

• Deep Neural Networks: Multilayer neural networks are well-suited for learning nonlinear relationships in text data. We trained a simple 3-layer feedforward network with 128 hidden units per layer using Keras.

• Convolutional Neural Networks (CNNs): CNNs use sliding filters to extract local features from input word embeddings, and have achieved state-of-the-art performance on many text classification tasks.[^8] We used a single 1D convolutional layer with 128 filters of size 5, followed by max pooling and 2 dense layers.

• Recurrent Neural Networks (RNNs): RNNs process sequences of words using a memory state to capture long-term dependencies. We used a single LSTM layer with 128 units followed by a dense output layer in Keras.

Results

We evaluated the performance of each embedding and model combination using 5-fold cross-validation on the tweet dataset. Here are the key results:

Embedding	Model	Accuracy	Precision	Recall	F1
TF-IDF	Logistic Regression	0.775	0.782	0.767	0.771
TF-IDF	Linear SVM	0.778	0.790	0.764	0.774
TF-IDF	Multinomial NB	0.744	0.767	0.731	0.734
Word2Vec	Logistic Regression	0.787	0.791	0.785	0.786
Word2Vec	Linear SVM	0.794	0.803	0.787	0.793
Word2Vec	Random Forest	0.768	0.757	0.761	0.758
Doc2Vec	Logistic Regression	0.818	0.822	0.814	0.817
Doc2Vec	Linear SVM	0.817	0.823	0.813	0.817
Doc2Vec	Gradient Boosting	0.804	0.818	0.798	0.807
LDA	Logistic Regression	0.705	0.736	0.674	0.692
CNN	Word2Vec	0.821	0.826	0.816	0.820
LSTM	Word2Vec	0.808	0.814	0.802	0.807

Key takeaways:

• Doc2Vec embeddings performed best across all models, showing a 3-5% absolute gain over Word2Vec and TF-IDF.
• Simple linear models like logistic regression and SVMs were highly competitive with more complex models like random forests and gradient boosting.
• Neural network models (CNNs and LSTMs) slightly outperformed linear models but took much longer to train.
• LDA topic embeddings performed significantly worse than the word/document embedding approaches.

The best performing models overall were logistic regression and linear SVMs trained on Doc2Vec embeddings, achieving around 82% accuracy on held-out data. Here is the normalized confusion matrix for the linear SVM:

We can see that the model performs well on negative and positive tweets, but struggles more with neutral tweets, often misclassifying them as negative. This highlights the challenge of identifying factual or objective statements that lack a clear sentiment signal.

Cross-Dataset Generalization

To test how well our trained models generalize to new data, we applied the best performing model (linear SVM with Doc2Vec) to two other airline tweet datasets:

1. Airline Sentiment 2 (2,363 labeled tweets)
2. Airline Sentiment 3 (3,483 labeled tweets)

The model achieved 75.6% and 79.1% accuracy on these datasets respectively, a drop of around 5-6% compared to the original dataset. This suggests some degree of overfitting to the specific language and topics of the training data.

To improve generalization, we could train the word/document embeddings on a much larger corpus of airline tweets, or use transfer learning from pretrained language models like BERT.^9 However, achieving very high accuracy on social media sentiment may be inherently difficult due to the noisiness and ambiguity of the data.

Conclusion and Future Directions

In this post, we conducted an extensive empirical comparison of machine learning approaches for tweet sentiment classification. We found that Doc2Vec embeddings outperformed other text representations, and that simple linear models were highly competitive with more sophisticated algorithms. However, achieving accuracy above 80% on this dataset was challenging due to the short, noisy nature of tweets and the class imbalance.

There are several promising directions for future research in this space:

1. Unsupervised or semi-supervised learning to leverage large amounts of unlabeled tweet data, e.g. pretraining embeddings or using data augmentation techniques.
2. Transformer-based models like BERT that capture richer linguistic information and context.
3. Multitask learning to jointly predict sentiment, emotions, sarcasm, etc. and improve generalization.
4. Incorporating additional tweet metadata like author profiles, threads, and social network structure.
5. Enabling interpretability and explanation of sentiment model predictions.
6. Evaluating model robustness to adversarial or out-of-domain examples.

With the rapid evolution of NLP techniques, and the growing importance of social media analysis, sentiment classification will likely remain an active and impactful research area for many years to come.

References

[^1]: Twitter Usage Statistics
[^2]: Challenges of Sentiment Analysis
[^3]: Analyzing Customer Sentiment in Airline Tweets
[^4]: Text Representations for Classification

[^7]: Latent Dirichlet Allocation
[^8]: Convolutional Neural Networks for Sentence Classification