Predicting the Toxicity of Comments Using Text Classification
The internet has enabled people worldwide to connect, share, and express themselves like never before. However, this open exchange has also given a platform to toxic behaviors like harassment, hate speech, and verbal abuse. Many online communities struggle to maintain civility in discussions while promoting free speech. Moderating user-generated content manually is increasingly infeasible due to the sheer scale. This is where text classification techniques can help, by automatically detecting toxic comments and flagging them for review or removal.
In this article, we‘ll dive into the fascinating world of toxic comment classification using machine learning. We‘ll start with an overview of text classification and its applications. Then we‘ll explore various approaches to building toxic comment classifiers, and walk through an example implementation. Finally, we‘ll discuss the challenges and considerations in this task, and look ahead to future research directions.
What is Text Classification?
Text classification is a common natural language processing (NLP) task that assigns predefined categories to text. Some popular applications include:
- Spam filtering: Is an email spam or not spam?
- Sentiment analysis: Is a movie review positive or negative?
- Topic labeling: What is a news article mainly about?
- Intent detection: Is the user intending to book a flight or check flight status?
The basic workflow of text classification is:
- Start with text documents labeled with categories
- Preprocess text – tokenization, removing stopwords/punctuation, stemming, etc.
- Extract numerical features from text – e.g. word counts, TF-IDF weights, word embeddings
- Train a machine learning model to map features to category labels
- Evaluate model on a test set
- Use model to predict categories for new unlabeled documents
This is an example of supervised machine learning, where we learn from labeled data to make predictions on unseen data. The two main components are feature extraction, which converts unstructured text to a structured numerical representation, and the classification algorithm that learns decision boundaries between classes.
Traditional machine learning approaches to text classification use bag-of-words features with models like Naïve Bayes, logistic regression, and support vector machines (SVMs). Neural networks are increasingly popular as they can learn feature representations automatically. Convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers are commonly used neural architectures for text.
The Problem of Toxic Comments Online
Toxic online comments, including hate speech, profanity, threats, and personal attacks, are a pervasive problem across social media, discussion forums, and news comment sections. A 2017 Pew Research Center survey found that 41% of American adults had experienced online harassment, with 66% witnessing it directed at others. Toxic comments make online spaces hostile and intimidating, silence marginalized voices, and polarize conversations.
Content moderation at scale is a huge challenge for online platforms. Facebook by some estimates makes about 300,000 content moderation decisions every day. Relying solely on human moderators is not sustainable. Automatic detection of toxic comments is crucial to maintaining healthy online communities.
However, identifying toxic comments is far from a straightforward task. Toxicity is a spectrum rather than a binary attribute. There are grey areas and edge cases, like sarcasm and reclaimed slurs. Toxicity is often context-dependent and requires nuanced understanding of language and social dynamics. Malicious users deliberately try to evade keywords filters. And of course, there are hundreds of languages to consider.
Approaches to Toxic Comment Classification
Early work on toxic comment classification used traditional ML models with simple features like bag-of-words and character n-grams. For example, Nobata et al. (2016) trained a regression model combining these features with linguistic and syntactic features. Davidson et al. (2017) found character n-grams to be most predictive with logistic regression and SVMs on several hate speech datasets.
More recent work has shifted to deep learning methods. CNN and RNN variants are popular for their ability to capture local and long-range semantic dependencies in text. Badjatiya et al. (2017) used a combination of CNN and Gradient-Boosted Decision Trees (GBDT). Agrawal & Awekar (2018) compared CNN, RNN, and Bi-LSTM architectures. Mishra et al. (2019) used a Bi-LSTM with attention. Graph convolutional networks have also been explored.
The rise of pre-trained language models like BERT and GPT-3 has enabled powerful transfer learning approaches. Rather than training from scratch, we can fine-tune a language model that has been pre-trained on a massive amount of unlabeled text. This is highly effective as the language model learns general features of language in pre-training, which can then be adapted to downstream tasks like toxic comment classification with relatively little labeled data. For example, Reimers (2019) fine-tuned BERT for toxic comment classification, outperforming previous methods.
Another consideration in toxic comment classification is that a comment may exhibit multiple types of toxicity simultaneously, like hate speech and profanity. This is a multi-label classification problem, as opposed to multi-class classification where each example has only one label. Multi-label problems require specialized loss functions and evaluation metrics. A simple approach is to train an array of binary classifiers, one for each label, known as binary relevance. More sophisticated methods, like classifier chains, take into account label dependencies.
Class imbalance is another challenge, as toxic comments are usually a small minority in real-world datasets. Oversampling the minority class, undersampling the majority class, and generating synthetic examples are strategies to handle this. At the algorithm level, adjusting class weights and using focal loss can help models focus on difficult examples.
Text preprocessing is also an important consideration. As toxic comments often contain misspellings, slang, and non-standard grammar, preserving these signals while still reducing noise is a delicate balance. Character-level models are less sensitive to spelling than word-based models. Emoji, hashtags, and username mentions also warrant special handling. Removing stopwords needs to be done carefully, as words like "you" can be strong indicators in personal attacks.
Example Implementation
Let‘s walk through an example implementation of toxic comment classification in Python. We‘ll use the Jigsaw Toxic Comments dataset from Kaggle, which contains Wikipedia talk page comments labeled for types of toxicity.
First, let‘s load and explore the data:
import pandas as pd
df = pd.read_csv(‘train.csv‘)
print(df.shape)
print(df.head())
The dataset has 6 toxicity labels: toxic, severe_toxic, obscene, threat, insult, and identity_hate. Let‘s check the class distributions:
print(df[df.columns[2:]].mean())
We can see the classes are quite imbalanced, with toxic being the most frequent at 9.6% and threat being the rarest at 0.3%.
Next let‘s preprocess the comment text. We‘ll use regex to strip HTML tags, URLs, and punctuation, lowercase, and tokenize into words:
import re
def preprocess(text):
text = re.sub(r‘<.*?>‘, ‘‘, text) # Remove HTML tags
text = re.sub(r‘https?://\S+|www.\S+‘, ‘‘, text) # Remove URLs
text = re.sub(r‘[^\w\s]‘, ‘‘, text) # Remove punctuation
text = text.lower() # Convert to lowercase
words = text.split() # Split into words
return words
df[‘words‘] = df[‘comment_text‘].apply(preprocess)
Now let‘s split the data into train and test sets:
from sklearn.model_selection import train_test_split
X = df[‘words‘] y = df[df.columns[2:]] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
For feature extraction, we‘ll use TF-IDF weighted word n-grams:
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(tokenizer=lambda x: x, preprocessor=lambda x: x, ngram_range=(1,2))
X_train = vectorizer.fit_transform([‘ ‘.join(words) for words in X_train])
X_test = vectorizer.transform([‘ ‘.join(words) for words in X_test])
Finally, let‘s train a logistic regression classifier for each toxicity label:
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
clf = OneVsRestClassifier(LogisticRegression())
clf.fit(X_train, y_train)
y_pred = clf.predict_proba(X_test)
print(‘Test ROC AUC scores:‘)
for i, label in enumerate(df.columns[2:]):
print(f‘{label}: {roc_auc_score(y_test[label], y_pred[i])}‘)
The ROC AUC scores look decent for a first attempt, with toxic at 0.97 and the rest in the 0.85-0.95 range. Of course, this is a basic implementation and there are many possible improvements, like using a neural model, fine-tuning a pre-trained language model, or ensembling multiple models. The purpose here is to illustrate the general workflow.
Challenges and Considerations in Toxic Comment Classification
While we‘ve made significant strides in toxic comment classification, many open challenges remain. Fundamentally, toxicity is a fuzzy concept that is difficult to precisely define. Not everyone may agree on what crosses the line. Toxicity judgments can be highly subjective and contextual.
Sarcasm and humor are notoriously hard for AI to detect, as they often rely on subtle linguistic cues and world knowledge. A comment that would be inoffensive literally may be toxic if sarcastic, and vice versa. Quotations and discussion of toxic language also complicate the task.
As soon as automated systems are deployed, malicious actors work to circumvent them. Intentional misspellings, word obfuscation, and dogwhistle terms are common evasion tactics. Keeping models up to date with rapidly evolving language is an arms race.
Ensuring fairness and mitigating unintended bias are critical challenges. Models can pick up on spurious correlations and perpetuate societal biases encoded in training data. For example, a model may associate toxicity with dialects or names of frequently attacked groups. Careful scrubbing of training data, bias auditing, and diverse oversight are essential. Microsoft‘s infamous Tay chatbot is a cautionary tale of what can go wrong.
Effective multilingual toxic comment classification is also a key challenge and area of research. Directly translating non-English text to English introduces noise, while obtaining large labeled datasets for all languages is prohibitively expensive. Zero-shot and few-shot cross-lingual transfer approaches offer promising solutions.
Future Directions
Toxic comment classification is an active research area with many interesting future directions. One is leveraging user and community information for context, like user‘s comment history, replies, upvotes/downvotes, and subreddit properties in the case of Reddit. Another avenue is federated learning to train models on sensitive user data while preserving privacy.
Multimodal models that incorporate audio, images, and video alongside text are also a promising direction, as toxicity can span multiple modalities. And as virtual and augmented reality spaces become increasingly popular, detecting toxicity in immersive settings will pose new challenges.
Responsible Deployment
When deploying toxic comment classifiers in the real world, we have a responsibility to carefully consider the potential impacts and failure modes. No model is perfect, and false positives can inadvertently censor non-toxic speech while false negatives can allow real harm. Models should not be the ultimate arbiters, but rather tools to assist human moderators.
Transparency about the use of automated moderation systems, including their capabilities and limitations, is important to maintain user trust. Users should have clear recourse if they feel unfairly censored. And of course, automated systems are not a substitute for comprehensive platform policies and efficient reporting and appeal processes.
It‘s also critical that diverse teams are involved in the development and oversight of these systems to avoid blind spots. Lived experience with the harms of toxic speech should inform this work. Regularly scheduled external audits can also help catch unintended model behaviors.
Conclusion
Text classification techniques have immense potential to help combat the scourge of toxic comments online and make the internet a safer, more inclusive space. However, this is a complex sociotechnical challenge that requires thoughtful solutions. Researchers and practitioners have made exciting progress, with sophisticated language models achieving impressive performance. But we must be ever vigilant of the limitations and potential pitfalls.
Toxic language is a reflection of deeper societal ills, and AI alone will not solve the problem. We need a holistic approach that includes platform design, content policies, social norms, and digital literacy education. With responsible innovation and proactive efforts from all stakeholders, we can work towards a vision of the internet where everyone can engage freely without fear of abuse. Automated toxic comment classification will be a key part of the solution, helping maintain the health of online communities at scale.
