# Leveraging Machine Learning to Detect and Combat Online Sexual Harassment

- Canonical: https://33rdsquare.com/classifying-sexual-harassment-using-machine-learning/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

The rise of social media and online communication has led to unprecedented opportunities for connection and self-expression. However, it has also enabled novel forms of abuse and harassment to proliferate. Online sexual harassment has emerged as a pervasive issue, causing serious harm to victims and chilling online discourse.

A 2017 [Pew Research Center study](https://www.pewresearch.org/internet/2017/07/11/online-harassment-2017/) found that 41% of Americans have personally experienced online harassment, with 18% subjected to severe behaviors like stalking, physical threats, and sexual harassment. The burden of this abuse is not evenly distributed – women, minorities, and younger internet users are disproportionately targeted.

A [2020 Plan International survey](https://plan-international.org/publications/freetobeonline/) of 14,000 girls across 22 countries found that 58% had experienced online harassment, with 19% changing or closing their social media accounts due to abuse. The consequences can be devastating, ranging from psychological trauma and self-censorship to doxing and real-world violence.

Manually moderating online spaces is an insurmountable challenge given the scale of user-generated content. On Facebook alone, [over 100 billion pieces of content are posted each day](https://www.facebook.com/business/news/insights/protecting-people-from-online-abuse). This is where machine learning can play a pivotal role, offering the ability to automatically detect abusive patterns and language at scale.

## Categories and Challenges of Online Sexual Harassment

Online sexual harassment can take many forms, ranging from unwanted sexual advances and crude jokes to revenge porn and rape threats. Some common categories include:

- Unwanted sexual attention: Repeated unsolicited sexual messages, images, or advances
- Gender-based harassment: Sexist slurs, attacks, and stereotypes targeting gender
- Sexual coercion: Pressuring for sexual content/favors, often via blackmail
- Violation of privacy: Sharing private sexual images/information without consent
- Sexual assault threats: Graphic threats of sexual violence, often with personal details

This diversity of behaviors poses challenges for machine learning models. There is no universal definition of what constitutes sexual harassment – it depends heavily on context, power dynamics, and individual perceptions. Crude language may be acceptable in certain online communities but abusive in others. Harassers may intentionally use coded language, sarcasm, and obfuscation to avoid detection.

Privacy concerns also introduce friction. Users may feel violated if their private conversations are screened for abuse. Malicious actors may attempt to weaponize reporting systems to censor others. Marginalized groups may face disproportionate scrutiny. Navigating this landscape requires thoughtful content policies, user controls, and human appeals processes.

## Machine Learning Approaches for Harassment Detection

At its core, online sexual harassment detection is a multi-class text classification problem. Given a unit of text (e.g. post, comment, message), the goal is to predict whether it falls into one or more predefined harassment categories. Both classical machine learning models and deep learning approaches have been applied to this task.

Some widely used classical ML algorithms for text classification include:

- Naive Bayes: A probabilistic model that predicts class probabilities based on word frequencies. While simple, Naive Bayes can perform well with appropriately engineered features.
- Support Vector Machines (SVM): Tries to find the maximum-margin hyperplane separating classes in a high-dimensional feature space. SVMs are effective for high-dimensional sparse data.
- Logistic Regression: Learns a linear decision boundary between classes by optimizing log likelihood. Logistic regression is computationally efficient and interpretable.
- Random Forests: An ensemble of decision trees trained on random subsets of features. Random forests reduce overfitting and can learn nonlinear decision boundaries.

In recent years, deep learning models have achieved state-of-the-art performance on many natural language processing tasks, including abusive language detection. Some popular architectures include:

- Convolutional Neural Networks (CNNs): Apply learned convolutional filters to extract local features from text. CNNs can capture important keywords and phrases.
- Recurrent Neural Networks (RNNs): Process text sequentially, maintaining a hidden state to capture long-term dependencies. LSTMs and GRUs are variants that mitigate vanishing gradients.
- Transformers: Use attention mechanisms to model pairwise interactions between words. Transformers can capture long-range context and have achieved state-of-the-art results on many tasks.

A key enabler of deep learning approaches has been the development of large-scale pre-trained language models like [BERT](https://arxiv.org/abs/1810.04805), [GPT](https://openai.com/blog/better-language-models/), and [XLNet](https://arxiv.org/abs/1906.08237). By pre-training on massive unlabeled text corpora, these models learn rich semantic representations that can be fine-tuned for downstream tasks like harassment detection with limited labeled data.

Some important design decisions when building harassment detection models include:

- Text Preprocessing: Cleaning raw text by removing HTML tags, links, and special characters. Tokenization into words, word-pieces, or characters. Handling case, punctuation, stop words, and stemming.
- Feature Extraction: Mapping text to numerical representations like bag-of-words, n-grams, or word embeddings. Syntactic features like POS tags and dependency relations. Sentiment and emotional tone. Domain-specific features like user metadata.
- Handling Class Imbalance: Real-world harassment datasets often have far fewer positive examples. Techniques like oversampling, undersampling, and data augmentation can prevent models from simply predicting the majority class.
- Model Interpretability: Understanding why models make certain predictions is crucial for detecting bias and improving performance. Techniques like LIME, SHAP, and attention visualization can explain individual predictions.
- Performance Metrics: Accuracy can be misleading for imbalanced datasets. Precision, recall, F1 score, and ROC AUC are better metrics. Confusion matrices help analyze error patterns.

## Real-World Deployment Challenges and Considerations

Developing accurate harassment detection models is only part of the battle. Responsibly deploying these systems involves grappling with a number of sociotechnical challenges:

- Defining Clear Policies: Every online community must clearly define what types of content are unacceptable, ideally with specific examples. Users need channels to report violations and appeal decisions. Policies may need customization for different contexts.
- Handling Edge Cases: No model is perfect. Automated systems must be able to handle sarcasm, humor, news discussions of harassment, and other edge cases without unduly censoring benign speech. Human review is essential for resolving ambiguous cases.
- Preventing Adversarial Attacks: Determined harassers may attempt to evade detection through lexical variation, intentional misspellings, and coded language. Models need to be robust to these attacks. Adversarial training can help harden models.
- Detecting Coordinated Campaigns: Some of the most severe harassment takes the form of coordinated "brigading" and "dogpiling" across multiple platforms. Detecting these emerging campaigns requires cross-platform information sharing and moderation partnerships.
- Transparency and Oversight: Automated moderation systems can have significant consequences for users‘ speech rights. Platforms must provide transparency into how these systems operate, including accuracy metrics and error rates. Third-party audits and oversight are important accountability mechanisms.
- Protecting Moderator Wellbeing: Commercial content moderators are routinely exposed to disturbing content, leading to trauma and PTSD. Automated systems can help reduce this exposure, but human moderators still need robust psychological support and counseling.

## Research Frontiers and Future Directions

Online harassment is a complex, evolving challenge that will require ongoing research to combat. Some promising areas for future work include:

- Multimodal Models: Harassers are increasingly communicating through images, video, and other non-text modalities. Models that can jointly reason over multiple modalities are an important research direction.
- Few-Shot Learning: With rapidly evolving online communities, new harassment terms and tactics are constantly emerging. Few-shot learning techniques that can quickly adapt to new types of abuse from limited examples will be essential.
- Knowledge-Enhanced Models: Incorporating real-world and community-specific knowledge into models, such as slang dictionaries, named entities, and conversation histories. Knowledge graphs and databases can provide valuable context.
- Cross-Lingual Models: Much online harassment occurs in languages other than English. Developing models that can operate across multiple languages and transfer knowledge between them is crucial for global impact.
- Human-AI Collaboration: Despite rapid advances, AI is not yet able to replace human judgment. Designing effective interfaces and workflows that empower human moderators to make nuanced decisions and provide feedback to improve models over time is a key challenge.

Ultimately, online sexual harassment is a societal problem that cannot be solved by technology alone. While AI will play an increasingly important role in triaging content and identifying abuse at scale, it must be thoughtfully combined with human moderation, clear platform policies, and robust user controls and reporting flows.

We must also recognize that online harassment is a reflection of broader offline power dynamics and inequities. Truly preventing harassment will require cultural and structural changes to address the root causes of abuse. This includes education on digital citizenship, promoting healthier masculine norms, and elevating voices of women and marginalized groups to shift power dynamics.

As an AI/ML community, our role is to develop ever more effective tools for protecting online spaces and empowering human moderators. By advancing the state-of-the-art in harassment detection and collaborating closely with impacted communities, we can help create a safer, more inclusive internet for all. It is a difficult challenge, but one we must rise to meet.

---

Source: [Leveraging Machine Learning to Detect and Combat Online Sexual Harassment](https://33rdsquare.com/classifying-sexual-harassment-using-machine-learning/)