WhatsApp Group Chat Analysis with Python: Uncover Hidden Insights

WhatsApp has taken the world by storm, becoming the go-to messaging app for over 2 billion users worldwide. On average, users spend around 195 minutes per week chatting on WhatsApp. With so much engagement, WhatsApp group chats can offer a treasure trove of interesting insights.

In this tutorial, we‘ll walk through how to use Python to analyze your WhatsApp group chats. By applying data science techniques, you can uncover fascinating statistics and trends about your group‘s dynamics and communication patterns. Let‘s dive in!

Setting Up Your Analysis Environment

To get started, you‘ll need to install a few handy Python libraries:

• re (regular expressions) – for extracting data from the chat text file
• pandas – for storing and manipulating the data
• matplotlib, seaborn, plotly – for creating visualizations
• numpy – for numerical computing
• datetime – for handling date and time data
• emoji – for detecting emojis in messages
• wordcloud – for generating word clouds
• heatmap – for visualizing activity levels
• nltk – for advanced text processing

You can install these with pip:

pip install regex pandas matplotlib seaborn numpy datetime emoji wordcloud heatmap plotly nltk

Collecting Your WhatsApp Chat Data

WhatsApp allows you to export a group‘s chat history as a text file. To do this:

1. Open the group chat
2. Tap the group subject
3. Scroll down and tap "Export Chat"
4. Choose "Without Media"
5. Save the chat text file

Now you have your raw chat data ready for analysis. The text file will contain lines with date, time, author and message text like this:

1/4/23, 9:05 AM - John: Hey everyone!
1/4/23, 9:06 AM - Jane: Hi John!
1/4/23, 9:07 AM - John: <Media omitted>

Extracting Structured Data with Regex

To analyze the chat, we first need to parse the text file and extract the key pieces of data into a structured format. We‘ll use regular expressions (regex) to find the date, time, author and message components on each line.

Here‘s a Python function that takes a line of text and returns the extracted data:

def getDataPoint(line):
    splitLine = line.split(‘ - ‘) 
    dateTime = splitLine[0]
    date, time = dateTime.split(‘, ‘)
    message = ‘ ‘.join(splitLine[1:])

    if FindAuthor(message):
        splitMessage = message.split(‘: ‘)
        author = splitMessage[0]
        message = ‘ ‘.join(splitMessage[1:])
    else:
        author = None

    return date, time, author, message

This uses the FindAuthor function to detect if the message contains an author name based on some regex patterns:

def FindAuthor(s):
    patterns = [
        ‘([w]+):‘,                        # First Name
        ‘([w]+[s]+[w]+):‘,              # First Name + Last Name
        ‘([w]+[s]+[w]+[s]+[w]+):‘,    # First Name + Middle Name + Last Name
        ‘([+]d{2} d{5} d{5}):‘,         # Mobile Number (India)
        ‘([+]d{2} d{3} d{3} d{4}):‘,   # Mobile Number (US)  
        ‘([w]+)[u263a-U0001f999]+:‘,    # Name and Emoji        
    ]
    pattern = ‘^‘ + ‘|‘.join(patterns)
    result = re.match(pattern, s)
    if result:
        return True
    return False

We‘ll run this function on every line of the chat file and store the extracted data in a list:

parsedData = [] 

conversationPath = ‘WhatsApp Chat Group.txt‘ 

with open(conversationPath, encoding="utf-8") as fp:
    fp.readline() 
    messageBuffer = [] 
    date, time, author = None, None, None
    while True:
        line = fp.readline()
        if not line: 
            break
        line = line.strip()
        if startsWithDateAndTime(line):
            if len(messageBuffer) > 0:
                parsedData.append([date, time, author, ‘ ‘.join(messageBuffer)])
            messageBuffer.clear()
            date, time, author, message = getDataPoint(line)
            messageBuffer.append(message)
        else:
            messageBuffer.append(line)

Now we have neatly parsed out the data we need for analysis into the parsedData list.

Storing Chat Data in a DataFrame

To make it easier to work with this data, let‘s load it into a pandas DataFrame:

import pandas as pd

df = pd.DataFrame(parsedData, columns=[‘Date‘, ‘Time‘, ‘Author‘, ‘Message‘])
df["Date"] = pd.to_datetime(df["Date"])

Before starting our analysis, we‘ll do some data cleaning:

df = df.dropna()
df = df.reset_index(drop=True)

This removes any rows with null values and resets the index.

We can also add a column for the day of the week:

import calendar

weekdays = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]

df[‘Weekday‘] = df[‘Date‘].dt.weekday
df[‘Day‘] = df[‘Weekday‘].apply(lambda x: calendar.day_name[x])

Our DataFrame is now ready for analysis! Here‘s a sample of what it looks like:

         Date      Time           Author                 Message Weekday       Day
0  2023-01-02  12:16 AM           Aditya              Hey there!       0    Monday   
1  2023-01-02  12:25 AM            Sonia                    Hi!       0    Monday
2  2023-01-02  12:30 AM           Aditya    <Media omitted>       0    Monday

Extracting High-Level Statistics

To get an overview of the group‘s activity, we can calculate some basic metrics:

total_messages = df.shape[0]
media_messages = df[df[‘Message‘] == ‘<Media omitted>‘].shape[0]

print(f‘Total messages: {total_messages}‘)
print(f‘Media messages: {media_messages}‘)

This gives us the total number of messages and media messages shared in the chat. We can extend this to get more granular stats like:

• Total emojis sent
• Links shared
• Avg message length
• Avg words per message

Analyzing User Activity

To understand each user‘s engagement, let‘s look at their individual stats:

from collections import defaultdict
from urlextract import URLExtract

stats = defaultdict(lambda : {"messages": 0, "emojis": 0, "media":0, "links":0, 
                              "words":0}) 

url_extractor = URLExtract()

for _, row in df.iterrows():
    author = row[‘Author‘]

    stats[author]["messages"] += 1
    stats[author]["words"] += len(row[‘Message‘].split())
    stats[author]["media"] += 1 if row[‘Message‘] == ‘<Media omitted>‘ else 0
    stats[author]["emojis"] += emoji.emoji_count(row[‘Message‘])
    stats[author]["links"] += len(url_extractor.find_urls(row[‘Message‘]))

for author, data in stats.items():
    print(f‘*{author}*n‘)
    print(f‘- Total messages: {data["messages"]}‘)  
    print(f‘- Avg words per message: {round(data["words"] / data["messages"], 2)}‘)
    print(f‘- Media: {data["media"]}‘)
    print(f‘- Links: {data["links"]}n‘)

This loops through each message, updating the metrics for the respective author. We use the emoji library to count emojis and urlextract to detect URLs.

The results give us a good picture of who the most active users are and their messaging patterns.

Visualizing Top Users and Content

Let‘s generate some visualizations to easily see the most engaged group members and popular content.

First, a bar chart of the top 10 most active users:

import matplotlib.pyplot as plt

top10_authors = df[‘Author‘].value_counts().head(10)

plt.figure(figsize=(10,5))
top10_authors.plot.bar()
plt.xlabel("User") 
plt.ylabel("Number of messages")
plt.title("Top 10 most active users")
plt.xticks(rotation=45)
plt.show()

We can do similar plots for the top 10 emojis, words, and links shared. To find the most frequent words, we‘ll tokenize the messages and remove stopwords:

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

stop_words = set(stopwords.words(‘english‘))

word_freq = defaultdict(int)

for msg in df[‘Message‘]:
    for word in word_tokenize(msg):
        if word.lower() not in stop_words:
            word_freq[word.lower()] += 1

top10_words = pd.Series(word_freq).sort_values(ascending=False).head(10) 

plt.figure(figsize=(10,5))
top10_words.plot.bar()
plt.xlabel("Word")
plt.ylabel("Frequency") 
plt.title("Top 10 most frequent words")
plt.show()

A word cloud is another great way to visualize the most used words:

from wordcloud import WordCloud

word_cloud = WordCloud(width=800, height=800, 
                      background_color=‘white‘,
                      stopwords=stop_words, 
                      min_font_size=10).generate_from_frequencies(word_freq)

plt.figure(figsize=(10,10))
plt.imshow(word_cloud)
plt.axis("off")
plt.show()

Time-Based Chat Activity

When are users most active? Let‘s look at messaging trends by day of week and hour:

plt.figure(figsize=(12,6)) 
hourly_activity = df.groupby(df[‘Time‘].dt.hour).count()[‘Message‘]
hourly_activity.plot.bar()

plt.xticks(range(24))
plt.xlabel("Hour of day")
plt.ylabel("Number of messages")  
plt.title("Hourly user activity")
plt.show()

daily_activity = df.groupby(‘Day‘).count()[‘Message‘]

plt.figure(figsize=(12,6))
daily_activity.reindex(weekdays).plot.bar()

plt.xlabel("Day of week")
plt.ylabel("Number of messages")
plt.title("Daily user activity")  
plt.show()

The hourly chart shows us the group‘s peak times, while the daily chart reveals which days are busiest. This can help inform when to send important messages for maximum engagement.

Analyzing Monthly Trends

For longer-term trends, let‘s examine activity by month:

df[‘Month‘] = pd.to_datetime(df[‘Date‘]).dt.strftime(‘%b %Y‘)

monthly_activity = df.groupby(‘Month‘).count()[‘Message‘]

plt.figure(figsize=(35,10))
monthly_activity.plot.line(marker="o")

plt.xticks(rotation=45)
plt.xlabel("Month")
plt.ylabel("Number of messages") 
plt.title("Monthly activity trends")
plt.show()

This line chart spans the full history of the chat, revealing any notable peaks or dips in engagement over time. It can help gauge the group‘s overall health and spot key moments.

Heatmap of User Activity

Finally, let‘s put it all together with a heatmap showing user activity by hour and day:

pivot = df.pivot_table(index=‘Day‘, columns=df[‘Time‘].dt.hour, values=‘Message‘, aggfunc=‘count‘)

plt.figure(figsize=(12,8))
sns.heatmap(pivot, cmap="YlGnBu")

plt.xlabel("Hour")
plt.ylabel("Day")
plt.title("User activity heatmap")
plt.show()  

The darker squares represent higher levels of chat activity at certain times and days. This detailed view helps us understand the group‘s daily rhythm.

Conclusion and Next Steps

In this tutorial, we covered how to use Python to extract valuable insights from a WhatsApp group chat. From user engagement metrics to activity trends, visualizations paint a clear picture of the group‘s dynamics.

These techniques merely scratch the surface of what‘s possible with chat data. Other interesting analyses could include:

• Sentiment analysis – gauge the emotions and opinions expressed in messages
• Topic modeling – uncover key themes and subjects discussed
• Name entity recognition – identify mentioned people, places, organizations
• Interaction networks – map the flow of messages between users

WhatsApp chats are a rich but often untapped data source. Applying data science allows us to better understand communication patterns and make data-driven decisions to optimize engagement.

I hope this guide has inspired you to try analyzing your own WhatsApp groups. Feel free to adapt the code and experiment with new techniques. If you have any questions or insights to share, please leave a comment!