WhatsApp Chat Analysis: Gaining Insights from Conversations

WhatsApp has become an ubiquitous part of our daily lives, with over 2 billion users worldwide sending tens of billions of messages each day on the platform. For many of us, a significant portion of our personal and professional communication now happens through WhatsApp. Our chat histories tell the stories of our relationships, our interests, our highs and lows.

But have you ever wondered what insights are hidden within your WhatsApp conversations? What patterns would emerge if you analyzed the data? How well do you really know your own chat behavior and that of your closest contacts?

In this article, we‘ll embark on an exciting data analysis project to find answers to these questions. We‘ll collect real WhatsApp chat data, parse and process it using Python, and visualize the results to uncover fascinating insights. Finally, we‘ll deploy an interactive web app to allow anyone to analyze their own chat histories. Let‘s dive in!

The WhatsApp Chat Analysis Process

Carrying out a WhatsApp chat analysis project involves several key steps:

1. Exporting chat data from WhatsApp
2. Preprocessing and cleaning the exported chat data
3. Exploratory analysis and visualization
4. Generating insights and statistics
5. Developing a web app for others to analyze their chats
6. Deploying the app to the cloud

We‘ll now go through each of these steps in detail, explaining the technical aspects and showing relevant code snippets in Python. We‘ll be using popular data science libraries like Pandas, Matplotlib, Seaborn and more.

Step 1: Exporting Chat Data

WhatsApp allows you to export your chat history for any individual or group conversation. To do this from your phone:

1. Open the chat you want to export
2. Tap on the menu or settings icon
3. Look for ‘Export Chat‘ or ‘More‘
4. Choose ‘Without Media‘
5. Send the chat file to yourself via email or cloud storage

This will generate a text file with all the messages, timestamps and senders‘ names. Be sure to do this a few times for different chats to get a varied dataset for analysis.

Step 2: Preprocessing the Chat Data

The raw chat data is not yet in a format we can work with. We need to parse the text file and extract the key components of each message into a structured table. Here‘s how we can do that using Python Pandas:

import re
import pandas as pd

def preprocess(chat):
    pattern = ‘\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{2}\s-\s‘
    messages = re.split(pattern, chat)[1:]
    dates = re.findall(pattern, chat)

    df = pd.DataFrame({‘raw_message‘: messages}, columns=[‘raw_message‘])
    df[‘date‘] = pd.to_datetime(dates, format=‘%d/%m/%Y, %H:%M - ‘)
    df.rename(columns={‘date‘: ‘timestamp‘}, inplace=True)

    users = []
    messages = []
    for message in df[‘raw_message‘]:
        entry = re.split(‘([\w\W]+?):\s‘, message)
        if entry[1:]:
            users.append(entry[1])
            messages.append(entry[2])
        else:
            users.append(‘group_notification‘)
            messages.append(entry[0])

    df[‘user‘] = users
    df[‘message‘] = messages
    df.drop(columns=[‘raw_message‘], inplace=True)

    df[‘only_date‘] = df[‘timestamp‘].dt.date
    df[‘year‘] = df[‘timestamp‘].dt.year
    df[‘month_num‘] = df[‘timestamp‘].dt.month
    df[‘month‘] = df[‘timestamp‘].dt.month_name()
    df[‘day‘] = df[‘timestamp‘].dt.day
    df[‘day_name‘] = df[‘timestamp‘].dt.day_name()
    df[‘hour‘] = df[‘timestamp‘].dt.hour
    df[‘minute‘] = df[‘timestamp‘].dt.minute

    return df

This function takes the raw chat export file, uses regex to separate the timestamp, sender and message components, and returns a clean Pandas DataFrame with additional columns for the date parts. We can now move on to analyzing this structured data.

Step 3: Exploratory Analysis & Visualization

Now comes the fun part – exploring the chat data to find interesting insights! We‘ll compute various metrics and statistics, and visualize them using Matplotlib and Seaborn. Some key things we‘ll look at:

• Chat activity trends over time
• Most active days of the week and times of day
• Top users by number of messages sent
• Most frequently used words and emojis
• Sentiment analysis of messages

Here‘s some sample code to get the top 10 most active users in the chat:

import matplotlib.pyplot as plt

def most_active_users(df):    
    top10 = df[‘user‘].value_counts().head(10)

    plt.figure(figsize=(10,4))
    plt.xticks(rotation=‘vertical‘)

    plt.bar(top10.index, top10.values, color=‘darkblue‘)
    plt.title(‘Most Active Users‘)
    plt.xlabel(‘Users‘) 
    plt.ylabel(‘Number of Messages‘)

    return plt

And here‘s how we can visualize the chat activity as a timeline:

def plot_timeline(df):
    timeline = df.groupby([‘year‘, ‘month‘]).count()[‘message‘]            
    timeline = timeline.reset_index().rename(columns={‘message‘:‘n_messages‘})
    timeline[‘month‘] = timeline[‘month‘].apply(lambda x: calendar.month_abbr[x])
    timeline[‘time‘] = timeline[‘month‘] + ‘ ‘ + timeline[‘year‘].astype(str)

    plt.figure(figsize=(12,6)) 
    plt.plot(timeline[‘time‘], timeline[‘n_messages‘], color=‘skyblue‘)
    plt.xticks(rotation=‘vertical‘)
    plt.title(‘WhatsApp Chat Activity Timeline‘) 
    plt.xlabel(‘Month‘)
    plt.ylabel(‘Number of Messages‘)

    return plt

We can generate many more charts and tables to find interesting patterns like:

• Distribution of message lengths
• Messages per hour of day as a heatmap
• WordCloud of most frequent terms
• Top 10 emojis used
• Percent of messages that are multimedia, links, deleted, etc.

The possibilities for analysis are endless! Check out the full code in the Resources section below.

Step 4: Generating Insights & Statistics

Based on our analysis, we can compile a useful set of metrics that summarize the chat behavior:

• Total messages sent
• Total words sent
• Total unique words
• Media messages sent
• Links shared
• Average messages per day
• Average word count per message
• Most active date
• Least active date
• Overall sentiment score

It‘s interesting to compare these numbers across different chats to spot contrasts in behavior. For example, you‘ll likely find you use more emojis and images when chatting with close friends compared to work groups.

Step 5: Building a Web App for WhatsApp Chat Analysis

To make it easy for anyone to gain insights from their WhatsApp chats, let‘s build an interactive web app! We‘ll use Streamlit, a Python framework that makes it a breeze to create data apps.

Our app will allow users to:

1. Upload a WhatsApp chat export file
2. See a summary of key stats about the chat
3. View auto-generated charts showing trends and patterns
4. Customize the analysis based on a date range and set of users

Here‘s a simplified version of the Streamlit app code:

import streamlit as st
from process import preprocess
from visualize import plot_timeline, most_active_users

st.sidebar.title("WhatsApp Chat Analyzer")

uploaded_file = st.sidebar.file_uploader("Choose a WhatsApp chat export file")

if uploaded_file is not None:
    chat = str(uploaded_file.read(),"utf-8")
    df = preprocess(chat)

    st.header("Chat Statistics")

    messages_count = df.shape[0]
    words_count = df[‘message‘].str.split().str.len().sum()
    media_count = df[df[‘message‘] == ‘<Media omitted>‘].shape[0]

    col1, col2, col3 = st.columns(3)

    with col1:
        st.metric("Total Messages", messages_count)
    with col2:
        st.metric("Total Words", words_count)
    with col3:  
        st.metric("Media Messages", media_count)

    timeline_chart = plot_timeline(df)
    st.pyplot(timeline_chart)

    most_active_chart = most_active_users(df)
    st.pyplot(most_active_chart)

    user_list = df[‘user‘].unique()
    selected_users = st.sidebar.multiselect("Select users", user_list)
    start_date = st.sidebar.date_input("Start date", df[‘only_date‘].min())
    end_date = st.sidebar.date_input("End date", df[‘only_date‘].max())

    if selected_users:
        df = df[(df["user"].isin(selected_users)) & (df["only_date"] >= start_date) & (df["only_date"] <= end_date)]

This code sets up a sidebar for uploading a chat file and selecting users/dates. It then displays some high-level statistics and the timeline + most active users charts we created earlier. Finally, it allows filtering the data based on the sidebar inputs.

Check out the live WhatsApp Chat Analyzer app and play around with your own chats!

Step 6: Deploying the App on the Cloud

While you can run the Streamlit app locally on your computer, it‘s more useful if it‘s live on the internet for others to access. We‘ll deploy our app to Heroku, a popular cloud platform.

The steps are:

1. Install Heroku CLI and login
2. Create a requirements.txt file with the Python dependencies
3. Create a Procfile and setup.sh with the Streamlit setup instructions
4. Initialize a Git repository and commit the code
5. Create a new Heroku app and push the code to deploy

Refer to Heroku‘s documentation for the detailed commands. Once deployed, your WhatsApp Chat Analyzer will have a public URL that anyone can visit and use!

Conclusion and Further Exploration

In this article, we walked through the journey of analyzing WhatsApp chats using Python – from exporting chat data to generating insights to deploying an interactive web app. The code and techniques we covered are a great starting point for your own chat analysis projects.

But there‘s a lot more you can do to take your analysis to the next level:

• Add more types of visualizations, like word treemaps and network graphs
• Implement sentiment analysis to quantify emotions and toxicity in chats
• Train a text generation model on your chat data to auto-suggest responses
• Analyze chat behavior differences across generations, genders, geographies
• Integrate more chat platforms like Telegram, Facebook Messenger, Discord

The potential for NLP and data science in messaging is immense and only growing as we spend more of our lives online. Analyzing digital conversations gives us a powerful lens to understand human behavior and improve how we communicate and relate to each other.

I encourage you to try out the WhatsApp Chat Analyzer with your own chat histories and share what insights you discover! Feel free to extend the code and explore more advanced NLP techniques. If you build something cool, do share it with the world.

Happy analyzing!

Resources and References

• Complete code for the WhatsApp Chat Analyzer on GitHub
• Deployed version of the WhatsApp Chat Analyzer app
• WhatsApp official guide for exporting chats
• Streamlit documentation for building data apps
• Heroku guide for deploying Streamlit apps
• Pandas documentation for data analysis
• Matplotlib and Seaborn gallery of visualizations
• NLTK and SpaCy for advanced NLP techniques