Embedding Techniques for Text Classification using KNN in PySpark

Text data is ubiquitous in the world today, from web pages and documents to social media posts and product reviews. Extracting insights and meaning from this unstructured data is a key challenge in natural language processing (NLP) and machine learning. One fundamental task is text classification – assigning predefined categories to documents based on their content.

Some common applications of text classification include:

• Sentiment analysis – determining if a review or opinion is positive, negative or neutral
• Topic categorization – organizing articles into categories like sports, politics, entertainment, etc.
• Spam detection – identifying and filtering out unwanted email or messages
• Author attribution – determining the author of a given document

To apply machine learning algorithms like K-Nearest Neighbors (KNN) to text, we first need to convert the unstructured text into structured numerical vectors, a process known as text vectorization or embedding. In this article, we‘ll dive into different embedding techniques and walk through a detailed example of building a text classifier using KNN in PySpark.

K-Nearest Neighbors

KNN is a non-parametric supervised learning method used for classification and regression. It‘s based on a simple intuition – examples that are similar tend to have the same label. To make a prediction for a new data point, KNN finds the k most similar examples in the training set and returns the majority label (for classification) or mean value (for regression).

Despite its simplicity, KNN can be surprisingly effective and is commonly used as a benchmark. Some advantages of KNN are:

• Easy to understand and implement
• Makes no assumptions about the underlying data distribution
• Naturally handles multi-class problems
• Can be effective with relatively small training sets

However, KNN also has some important limitations:

• Computationally expensive for large datasets, since all pairwise distances must be calculated
• Sensitive to irrelevant or noisy features, which can dominate the distance calculations
• Requires storing the entire training set in memory
• Optimal choice of hyperparameters (k, distance metric) may not be obvious

Text Vectorization Methods

To apply KNN to text data, we first need to transform documents into fixed-length numerical feature vectors. Some common techniques include:

Bag-of-Words

Bag-of-words (BoW) represents a document as a vector of word counts, disregarding grammar and word order. The vocabulary is determined from the training corpus and each document is encoded by counting how many times each word appears.

For example, consider the sentences:

• "John likes to watch movies. Mary likes movies too."
• "John also likes to watch football games."

With a vocabulary of {John, likes, to, watch, movies, Mary, too, also, football, games}, the BoW vectors would be:

• [1, 2, 1, 1, 2, 1, 1, 0, 0, 0]
• [1, 1, 1, 1, 0, 0, 0, 1, 1, 1]

BoW is simple but can lead to high-dimensional sparse vectors for large corpora. Words are treated as atomic units, ignoring meaning and semantics.

TF-IDF

Term Frequency-Inverse Document Frequency (TF-IDF) is an extension of BoW that weights words by how frequently they appear in a document (TF) and how document-specific they are (IDF). The intuition is that words appearing often in a document but rarely in the overall corpus are more informative.

TF-IDF vectors are calculated as:

tfidf(t,d) = tf(t,d) * idf(t)

where

• tf(t,d) = (number of times term t appears in document d) / (total number of terms in d)
• idf(t) = log((total number of documents) / (number of documents with term t))

TF-IDF can help downweight frequent but uninformative words. However, it still represents documents as a bag of independent words.

Word Embeddings

Word embeddings are a family of techniques that learn dense vector representations of words, such that semantically similar words have similar vectors. They are typically trained on large unsupervised corpora in a self-supervised fashion.

Some popular word embedding algorithms are:

• Word2Vec – Trains shallow neural networks to predict a word from its context (skip-gram) or vice versa (CBOW). The learned weight matrices contain meaningful word vectors.

• GloVe – Learns word vectors by factoring the logarithm of the word-word co-occurrence count matrix.

• FastText – An extension of Word2Vec that incorporates subword information, allowing embedding of out-of-vocabulary words.

Word embeddings capture rich semantic and syntactic relationships, such as gender, tense, pluralization, etc. Embedding vectors are typically 100-500 dimensions. To obtain document vectors, the word vectors can be aggregated by averaging or weighted averaging using TF-IDF scores.

PySpark KNN Example

Now let‘s walk through an example of building a KNN text classifier in PySpark. We‘ll use the 20 Newsgroups dataset, a collection of ~20,000 documents partitioned into 20 topics.

First, load the data into a Spark DataFrame and split into train/test sets:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder \
    .appName("KNN 20 Newsgroups") \
    .getOrCreate()

train_df = spark.read.json("20news-bydate-train/*")    
test_df = spark.read.json("20news-bydate-test/*")

Next, extract TF-IDF vectors using the HashingTF and IDF functions in the pyspark.ml.feature module:

from pyspark.ml.feature import HashingTF, IDF

hashingTF = HashingTF(inputCol="text", outputCol="tf", numFeatures=1000)
tf_train = hashingTF.transform(train_df)
tf_test = hashingTF.transform(test_df)

idf = IDF(inputCol="tf", outputCol="tfidf")
idf_model = idf.fit(tf_train) 
tfidf_train = idf_model.transform(tf_train)
tfidf_test = idf_model.transform(tf_test)

Now we can train the KNN model using the KNNClassifier in pyspark.ml.classification. We specify the feature vector column, label column, and the optimal k found by hyperparameter tuning:

from pyspark.ml.classification import KNNClassifier

best_k = 30 # Found by grid search
knn = KNNClassifier(featuresCol="tfidf", labelCol="label", k=best_k) 
knn_model = knn.fit(tfidf_train)

Finally, we can evaluate performance on the test set:

predictions = knn_model.transform(tfidf_test)

from pyspark.ml.evaluation import MulticlassClassificationEvaluator

evaluator = MulticlassClassificationEvaluator(predictionCol="prediction", labelCol="label",
                                              metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("Test Accuracy = %g" % accuracy) 

This prints an accuracy of around 0.83, which is quite good for a simple model! We could further improve performance by tuning the number of hash buckets, trying different distance functions, or using word embeddings instead of TF-IDF. Some other tips:

• Preprocess text by lowercasing, removing punctuation/stopwords, stemming, etc.
• Use cross-validation to select the best model
• Experiment with different values of k – larger values tend to smooth out noise but have higher computation cost
• Consider approximate nearest neighbor methods to speed up search, e.g. Locality Sensitive Hashing
• Ensemble KNN with other models like naive Bayes or logistic regression

Conclusion

In this article, we discussed various techniques for representing text data as numerical vectors and demonstrated how to build a document classifier using KNN in PySpark. Key takeaways are:

• Text data must be transformed into vectors for machine learning, via methods like bag-of-words, TF-IDF, or word embeddings
• KNN is a simple instance-based learning algorithm that can be effective for text classification
• PySpark provides built-in tools for NLP pipelines that can scale to large datasets
• Experimenting with different embeddings and hyperparameters is important to get the best performance

With the explosive growth of text data, NLP and text mining skills are becoming increasingly valuable. Mastering core concepts like vectorization and classification is essential for any aspiring data scientist or machine learning engineer. Further topics to explore include document clustering, topic modeling, named entity recognition, sentiment analysis, and deep learning architectures like CNN and transformers. Happy learning!