Detecting Spam SMS Messages with Naive Bayes and SVM Classifiers

In today‘s digital age, unwanted spam messages are an annoying and pervasive problem. Beyond being a nuisance, spam SMS can also be a vehicle for scams, phishing attempts, and malware. Manually identifying and filtering out spam messages is time-consuming and impractical. Fortunately, machine learning techniques provide an automated way to classify SMS as spam or legitimate.

In this article, we‘ll dive into two popular machine learning algorithms for text classification – Naive Bayes and Support Vector Machines (SVM). We‘ll explore the intuition behind these models, preprocess a real-world SMS spam dataset, and implement the classifiers in Python. By the end, you‘ll have a solid understanding of how to build your own spam SMS detection system. Let‘s get started!

The Naive Bayes Classifier

Naive Bayes is a probabilistic machine learning algorithm based on applying Bayes‘ theorem with a strong assumption of independence between the features. For text classification, the features are the words in the document.

Bayes‘ theorem describes the probability of an event based on prior knowledge of conditions that might be related to the event. Mathematically, it states that the probability of A given B is equal to the probability of B given A times the probability of A, divided by the probability of B:

P(A|B) = P(B|A) * P(A) / P(B)

The "naive" in Naive Bayes comes from the assumption that the features (words) are conditionally independent from each other given the class. In other words, the presence or absence of a particular word does not influence the presence or absence of another word, given the SMS is spam or ham.

While this assumption is obviously not true in reality since words in sentences have order and relationships, Naive Bayes models still tend to perform surprisingly well in practice. They are fast, scalable, and easy to implement.

For text classification, we use the Multinomial Naive Bayes variant which models the frequencies of words. The algorithm calculates the probability of a document belonging to a class by multiplying the probabilities of each word given the class, with some smoothing to avoid probabilities of zero. The class with the highest probability is chosen as the predicted class.

One limitation of Naive Bayes is that it can struggle with imbalanced datasets. If there are very few examples of spam compared to ham SMS, the model may have trouble learning the patterns of spam. It also typically performs worse than discriminative models like SVM given enough training data.

Support Vector Machines

Support Vector Machines (SVM) are powerful and versatile supervised machine learning models used for both classification and regression. They are especially well-suited to high-dimensional data like text.

The core idea behind SVM is to find the hyperplane (a line in 2D, plane in 3D, etc.) that best separates the classes in feature space. The optimal hyperplane is the one that maximizes the margin – the distance between the hyperplane and the closest data points from each class, called the support vectors.

Mathematically, an SVM model represents the examples as points in space mapped so that the examples of the separate classes are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a class based on which side of the gap they fall.

In many real-world problems, the classes are not linearly separable in the original feature space. SVM handles this by using the kernel trick to implicitly map the inputs into high-dimensional feature spaces where a linear separation is possible.

The most common kernels are:

• Linear: no transformation, used when data is linearly separable
• Polynomial: transforms data into higher degree polynomial features
• Radial Basis Function (RBF): Gaussian kernel, maps data into infinite dimensional space

In practice, the linear kernel works well for sparse high-dimensional data like text, and is faster to train. The hyperparameter C controls the trade-off between smooth decision boundaries and correctly classifying the training examples.

SVMs have several advantages over Naive Bayes:

• They can learn complex non-linear decision boundaries
• They are robust to overfitting, especially in high dimensions
• They only rely on a small subset of the training data (the support vectors)

Some disadvantages are:

• They are slower to train, especially with large datasets
• They require careful tuning of hyperparameters like the kernel and regularization
• The learned model is a "black box" and not easily interpretable

SMS Spam Dataset

To see the classifiers in action, we‘ll use the SMS Spam Collection Dataset from the UCI Machine Learning Repository. This dataset contains 5,574 English SMS messages, each labeled as either "ham" (legitimate) or "spam". The messages were collected from various sources including personal messages, donations, and SMS message boards.

Here are the first few rows of the data:

ham Go until jurong point, crazy.. Available only in bugis n great world la e buffet… Cine there got amore wat…
ham Ok lar… Joking wif u oni…
spam Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C‘s apply 08452810075over18‘s
ham U dun say so early hor… U c already then say…
ham Nah I don‘t think he goes to usf, he lives around here though

The data is tab separated with the label (ham/spam) as the first column and the SMS text in the second column. The messages contain a lot of informal language, abbreviations, and slang which we‘ll need to preprocess.

Text Preprocessing

Machine learning models require numerical feature vectors, so we need to convert the raw text messages into a suitable representation. We‘ll use the common bag-of-words model where each unique word in the corpus becomes a feature.

The text preprocessing pipeline:

1. Tokenization – split each message into a list of words
2. Case normalization – convert all words to lowercase
3. Punctuation removal – remove punctuation and special characters
4. Stop word removal – remove common words like "a", "the", "in" that don‘t contain much discriminative information
5. Stemming/Lemmatization – reduce words to their base or dictionary form (e.g. "running" -> "run")

After preprocessing, we‘ll have a "clean" list of words for each message. We then convert each message into a numerical vector using the CountVectorizer in scikit-learn. This creates a sparse matrix where each row is an SMS message, each column is a unique word (unigram) from the corpus, and each entry is the count of that word in that message.

Model Training and Evaluation

With the data prepared, we‘re ready to train the models. We first split the data into training and test sets using an 80/20 split. This allows us to evaluate how well the models generalize to unseen data.

For Naive Bayes, we use the MultinomialNB class from scikit-learn. This implementation incorporates additive (Laplace/Lidstone) smoothing which helps prevent zero probabilities by adding a small constant to each word count.

For SVM, we use the LinearSVC class which is an efficient implementation of linear Support Vector Classification. The C hyperparameter is tuned using cross validation to find the optimal trade-off between misclassification and margin width.

To evaluate the trained models, we make predictions on the test set and compare them to the true labels. We‘ll use several standard metrics for binary classification:

• Accuracy: fraction of predictions that are correct
• Precision: fraction of positive predictions that are correct
• Recall: fraction of actual positives that are correctly predicted
• F1 score: harmonic mean of precision and recall

Comparing the performance of Naive Bayes and SVM on the spam SMS dataset, we see that both achieve very high accuracy over 97%. SVM slightly outperforms Naive Bayes across all metrics, with an F1 score of 0.97 compared to 0.96 for Naive Bayes.

This shows that for this particular task of spam SMS detection, a simple linear SVM is sufficient to achieve excellent performance. The Naive Bayes assumption of word independence also turns out to work quite well here.

Implementing in Python with scikit-learn

Here‘s the complete Python code to preprocess the data, train the models, and evaluate their performance using scikit-learn:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Load data 
df = pd.read_csv(‘spam.csv‘, sep=‘\t‘, header=None, names=[‘label‘, ‘text‘])

# Preprocess data
def preprocess(text):
    text = text.lower() 
    text = re.sub(r‘[^a-z ]‘, ‘‘, text)
    text = ‘ ‘.join([word for word in text.split() if word not in stopwords])
    text = ‘ ‘.join([stemmer.stem(word) for word in text.split()])
    return text

df[‘text‘] = df[‘text‘].apply(preprocess)    

# Create bag-of-words representation
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(df[‘text‘])
y = df[‘label‘]

# Split into train and test sets  
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Naive Bayes model
nb = MultinomialNB()
nb.fit(X_train, y_train)

# Train SVM model
svm = LinearSVC(C=1.0)  
svm.fit(X_train, y_train)

# Evaluate models on test set
nb_pred = nb.predict(X_test)
svm_pred = svm.predict(X_test)

print("Naive Bayes:")
print("Accuracy:", accuracy_score(y_test, nb_pred))  
print("Precision:", precision_score(y_test, nb_pred, pos_label=‘spam‘))
print("Recall:", recall_score(y_test, nb_pred, pos_label=‘spam‘))
print("F1 score:", f1_score(y_test, nb_pred, pos_label=‘spam‘))

print("SVM:") 
print("Accuracy:", accuracy_score(y_test, svm_pred))
print("Precision:", precision_score(y_test, svm_pred, pos_label=‘spam‘))  
print("Recall:", recall_score(y_test, svm_pred, pos_label=‘spam‘))
print("F1 score:", f1_score(y_test, svm_pred, pos_label=‘spam‘))

This script loads the spam SMS dataset, preprocesses the text, creates a bag-of-words representation, trains Naive Bayes and SVM models, and evaluates their performance on the test set. You can run it yourself to reproduce the results.

Conclusion and Future Work

In this article, we explored using Naive Bayes and SVM classifiers to detect spam SMS messages. We preprocessed the raw text data, transformed it into numerical features, trained the models, and evaluated their performance on a held-out test set.

Both models achieved very high accuracy, with SVM slightly outperforming Naive Bayes. This demonstrates the effectiveness of these classic machine learning techniques for text classification tasks.

However, there are some limitations and areas for future work:

• The bag-of-words representation ignores word order and context. More advanced techniques like word embeddings or deep learning models may be able to capture richer semantic information.
• The dataset is quite small by modern standards. Collecting more labeled examples, especially of spam messages, could help improve performance further.
• Spam messages often include obfuscated words, intentional misspellings, and non-textual content like links and images. Extracting features to capture these patterns could boost classification accuracy.
• In a production system, the model should be regularly updated as spammers adapt their techniques over time. Online learning and periodic retraining on new data can help the model keep up.

Despite these limitations, the strong performance of Naive Bayes and SVM show that they remain valuable tools in the machine learning practitioner‘s toolkit. For a straightforward binary text classification problem like spam detection, they can get you very far. I encourage you to try implementing these models yourself on other datasets and see what results you can achieve!