Twitter-Based Gender Classification Using Machine Learning

Social media platforms like Twitter have become a rich source of data for analyzing human behavior and demographics at a massive scale. An active area of research is automatically inferring attributes of users such as age, gender, location, political affiliation and more based solely on the content they post. These methods can help understand differences in how user groups engage with social media and can power more personalized user experiences.

In this post, we‘ll dive deep into the task of predicting the gender of Twitter users based on their tweets and profile information using machine learning. We‘ll walk through each step of the process, explore different approaches, and discuss important considerations when putting these systems into practice.

Problem Definition and Applications

Given a Twitter user‘s profile information and a sample of their tweets, the goal is to predict whether the user is male or female. Assuming gender is a binary attribute is itself a simplification, but we‘ll stick with that formulation for this post since most datasets are labeled that way.

Why is this an interesting and useful problem to solve? There are a few key applications:

• Demographic analysis: Gender classification can give a demographic breakdown of a brand or celebrity‘s audience, or show how men vs. women differ in their interests and opinions on various topics based on what they tweet about.
• Personalization: Knowing a user‘s gender can allow for more relevant content recommendations, product suggestions, and advertisements.
• Bias and fairness auditing: Are our models exhibiting gender bias? Are female users more likely to be misclassified? Measuring gender disparities can highlight issues to correct for.
• Addressing abuse: In some cases of online harassment, the gender of the users involved is a relevant factor when reviewing abusive behavior.

At the same time, it‘s critical to consider the ethical implications of automatically classifying personal attributes like gender. Users may feel it violates their privacy and could enable discrimination. We‘ll touch more on responsible use of these technologies later on.

Dataset and Preprocessing

There are a few publicly available datasets commonly used for Twitter gender classification. For this post we‘ll use the Twitter User Gender Classification dataset on Kaggle.

It contains 20,000 rows, each representing a different Twitter user. The useful fields are:

• description: the user‘s profile description
• text: a random tweet by the user
• gender: "male", "female", or "brand" (non-human)
• gender_confidence: confidence in the gender label

First, let‘s load the data and take a look:

import pandas as pd
data = pd.read_csv("gender-classifier-DFE-791531.csv", encoding="latin1")
data.head()

The first preprocessing steps are:

1. Remove rows with label "brand" since we only want human users
2. Remove rows with confidence below 1 to avoid noisy labels
3. Binarize "gender" labels to 0 for female and 1 for male
4. Concatenate "description" and "text" fields into a single "tweet" field to use as input

data = data[data.gender != "brand"]
data = data[data.gender_confidence == 1]
data["gender"] = data["gender"].map({"female": 0, "male": 1})
data["tweet"] = data["description"] + " " + data["text"] 
data = data[["tweet", "gender"]]

After this, we‘re left with 10,000 labeled examples. The classes are fairly balanced, with 52% female and 48% male.

Next we convert the raw text into features usable by ML models.

Feature Extraction

Our input is freeform text, but ML models need numeric feature vectors. There are a few ways we can represent tweets:

Bag-of-words: Represent each tweet by the counts of all words in the vocabulary, ignoring order. Simple but surprisingly effective for short documents.

n-grams: Similar to bag-of-words, but counts occurrences of phrases with n consecutive words. Captures some local word order.

Character n-grams: Same as n-grams but on the character level. Useful for detecting slang and misspellings common in tweets.

Tweet-level features: Counts of tweet-specific attributes like number of hashtags, mentions, links, and punctuation types. Can be combined with any of the above.

For this example we‘ll use a simple bag-of-words representation. The CountVectorizer in scikit-learn makes this easy:

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words="english", max_features=3000)
features = vectorizer.fit_transform(data["tweet"])
labels = data["gender"]

This converts each tweet to a vector with counts for the 3000 most common non-stopwords in the corpus. We can inspect what the model considers the most predictive words for each gender:

for gender, words in [("Female", vectorizer.inverse_transform(model.coef_  0)[0])]:
    print(f"{gender}: {‘ ‘.join(words)}")

Female: love my girl mom so happy 💕
Male: bro man dude the lol shit fuck

These word associations match common gender stereotypes. The model seems to be picking up on genuine linguistic differences between genders, but some of it may reflect bias in labeling rather than how users actually identify.

Model Training and Evaluation

With our features and labels in hand, we‘re ready to train some models! Let‘s compare a few options:

Naive Bayes: A probabilistic classifier that assumes features are independent. Works well with small data and is quick to train.

Logistic Regression: A linear model that learns a weight for each feature to predict the log odds ratio of the positive class. Weights can be inspected to see impacts of each feature.

Support Vector Machine (SVM): Tries to find the maximum margin hyperplane that best separates the classes. Performs well in high dimensions.

Random Forest: An ensemble of decision trees fit on random subsets of data. Provides good accuracy and can measure feature importances.

We‘ll use 10-fold cross-validation to get an unbiased estimate of each model‘s performance and measure precision, recall, and F1 in addition to accuracy to account for class imbalance.

from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate
models = [
("Naive Bayes", MultinomialNB()),
("Logistic Regression", LogisticRegression(max_iter=200)),
("Linear SVM", LinearSVC()),

("Random Forest", RandomForestClassifier(n_estimators=200))
]
scores = []
for name, model in models:
scores.append(cross_validate(model, features, labels, cv=10, 
scoring=["accuracy", "precision", "recall", "f1"]))
print("Mean 10-fold scores:")
for name, score in zip(names, scores):
print(f"{name}: accuracy={score[‘test_accuracy‘].mean():.3f} precision={score[‘test_precision‘].mean():.3f} " \
f"recall={score[‘test_recall‘].mean():.3f} f1={score[‘test_f1‘].mean():.3f}")

Mean 10-fold scores:
Naive Bayes: accuracy=0.635 precision=0.627 recall=0.667 f1=0.643
Logistic Regression: accuracy=0.679 precision=0.682 recall=0.670 f1=0.673
Linear SVM: accuracy=0.679 precision=0.675 recall=0.684 f1=0.678
Random Forest: accuracy=0.681 precision=0.686 recall=0.666 f1=0.674

The logistic regression, SVM, and random forest all achieve around 68% accuracy, a few points higher than Naive Bayes. Random forests tend to work better with more data. Since we have a fairly small dataset, the linear models do about as well.

Let‘s choose the logistic regression for simplicity and inspect its coefficients:

model = LogisticRegression(max_iter=200).fit(features, labels)
sorted(list(zip(model.coef_[0], vectorizer.get_feature_names())),
key=lambda x: -abs(x[0]))[:10]

[(‘husband‘, -0.945),
(‘love‘, -0.654),
(‘my‘, -0.634),
(‘family‘, -0.542),
(‘wife‘, -0.493),
(‘the‘, 0.455),
(‘founder‘, 0.452),
(‘dad‘, 0.422),
(‘man‘, 0.413),
(‘father‘, 0.400)]

The most important features are intuitive – words like "love", "family", and "husband" are associated with female users, while "man", "dad" and "father" lean male. Tweets alone provide a surprising amount of signal into user demographics.

Error Analysis

To better understand our model‘s limitations, let‘s look at some examples it misclassified. Here are a few female users the model thought were male:

• Society has a problem with female aggression but I‘m a bad ass bitch
• I‘m an engineer pursuing a PhD and like whiskey so maybe I‘m not the typical female
• I hate all these sexist jokes about women belonging in the kitchen. Just kidding, I love the kitchen

And here are some male users misclassified as female:

• My girlfriend is the sweetest! Love you babe 😘
• Fellas, get you a girl that can cook because these microwave dinners aren‘t cutting it
• Sure I‘m a sensitive guy, but I still love football and video games

We can see the model struggles with users who don‘t fit neatly into gender stereotypes – women who use more aggressive language or work in male-dominated fields, and men who express more emotional sentiments.

It also fails on examples relying heavily on sarcasm, humor, or references to other people like girlfriends. Detecting these nuances is challenging from isolated snippets of text.

There‘s an inherent limitation in relying on stereotypical word associations to determine gender – it doesn‘t account for the full diversity in how people express gender identity. The task formulation itself reflects reductive assumptions about binary gender.

Fairness and Ethical Considerations

On the technical side, there are issues of fairness and robustness to consider. We saw the model picks up on stereotypical word associations which may reflect bias in training data rather than real differences in language use. This can harm users who get misclassified, say if their content is censored or they receive less relevant recommendations.

There‘s also a feedback loop risk if these predictions are used to retrain the model or make decisions that impact what the user sees. The model may get more confident in its stereotypes rather than capturing the true diversity of gendered behavior.

On the product side, is gender inference something users expect and consent to? How can predictions be used responsibly, without limiting user autonomy? There are valid use cases like analytics and anti-abuse efforts, but the privacy tradeoffs must be considered. Features like this are best used in aggregate, not to target individuals.

More fundamentally, classifying gender as binary is itself a simplification that fails to recognize the many expressions of gender identity. Why are we ascribing gender to users in the first place? Any use case should be scrutinized to ensure gender data is truly needed.

When building ML systems that impact peoples‘ online experiences, we must be thoughtful about what assumptions we bake in. Models will learn whatever biases exist in the data and goals we provide. It‘s crucial to go beyond optimizing accuracy and consider the social context in which these systems are deployed.

Conclusion

We‘ve walked through building a machine learning pipeline to classify the gender of Twitter users based on the content of their tweets. With a dataset of 10K users, a simple logistic regression model achieved 68% accuracy using bag-of-words features. Inspecting the heavily weighted words surfaced some common gendered associations.

Error analysis revealed the model struggles with sarcasm, humor, and users who defy stereotypes. This highlights the risk of models perpetuating reductive notions of gender.

There are important ethical considerations to using this technology. Models can reflect biases in training data and their predictions can limit user autonomy if applied without consent. The binary gender classification setup fails to capture the full spectrum of gender identity.

Overall, building demographic inference models is a powerful application of NLP, but one that requires care. We must go beyond accuracy and consider fairness, robustness, and responsible design when deploying these systems. Only by recognizing the social context can we build equitable ML systems.