Sentiment Analysis on Flipkart Product Reviews Dataset

In today‘s digital age, e-commerce has revolutionized the way we shop for products and services. Online marketplaces like Flipkart allow customers to easily browse a wide selection of offerings, compare prices, and make purchases with a few clicks. Crucially, these platforms also enable customers to share their opinions and experiences by writing product reviews.

This user-generated review text is a goldmine for understanding customer sentiment – but the scale is far too large for any human to process. That‘s where sentiment analysis comes in. By applying natural language processing and machine learning techniques, we can automatically analyze thousands or millions of product reviews and gauge whether customers are satisfied or dissatisfied, and why.

In this post, we‘ll walk through an example of performing sentiment analysis on a dataset of Flipkart product reviews. We‘ll cover the key steps from data cleaning through model training and evaluation. Finally, we‘ll discuss some valuable insights and applications that sentiment analysis unlocks for both e-commerce businesses and customers. Let‘s dive in!

The Flipkart Reviews Dataset

Our dataset consists of 10,000 reviews posted by Flipkart customers for products across various categories including electronics, clothing, home goods, and more. The data is structured as follows:


review_id - unique identifier for each review
product_id - unique identifier for each product
reviewer_id - unique identifier for each reviewer
reviewer_name - name of the reviewer
review_title - title of the review
review_text - text of the review
review_rating - rating assigned by reviewer on a 1-5 scale


The review_text and review_rating columns will be our primary focus for sentiment analysis. The ratings provide a convenient source of sentiment labels, while the review text is what we‘ll actually be analyzing to predict those sentiment labels.

Data Preprocessing

Before we can train a sentiment analysis model, we need to clean and preprocess the text data to remove noise and convert it into a standard machine-readable format. Some common text preprocessing steps include:

1. Removing punctuation and special characters

Commas, periods, exclamation points and other punctuation marks are removed since they don‘t add much sentiment signal. We use regex substitution to strip these out.

2. Converting to lowercase

We convert all text to lowercase so that "Good" and "good" are treated the same way. This reduces the vocabulary size and helps the model focus on the important content words.

3. Removing stopwords

Stopwords are common words like "the", "and", "a" that appear frequently but don‘t convey much meaningful information. We can remove them to reduce the dimensionality of the data. The NLTK library provides a built-in list of English stopwords.

4. Stemming/Lemmatization

Stemming and lemmatization are techniques to standardize different inflected forms of words. For example, "look", "looking", "looked", "looks" would all be shortened to their common root "look". This aggregates semantically similar words to reduce complexity. Popular algorithms include the Porter Stemmer and WordNet Lemmatizer.

After applying these steps to preprocess each review text, we‘re ready to extract numerical features that can be fed into a machine learning model.

Feature Extraction

Since machine learning models take numerical vectors as input, we need to convert the preprocessed text into a numerical representation. Some options include:

1. Bag-of-Words

In this simple representation, we build a vocabulary of all unique words across the corpus, and represent each review as a vector of word occurrence counts. The major shortcoming is that it ignores word order and contextual information.

2. TF-IDF

Term Frequency-Inverse Document Frequency is an extension of Bag-of-Words that weights words based on their frequency within and across documents. This helps identify words that are highly informative for each review.

3. Word Embeddings

Word embedding models like Word2Vec and GloVe learn to map semantic meaning into a compact vector space, such that similar words have similar vectors. By using pretrained embeddings, we can represent reviews as an average or sequence of word vectors that preserve semantic relationships. This is state-of-the-art for many NLP tasks.

For our Flipkart dataset, let‘s use a TF-IDF representation to capture the most salient keywords in each review. We‘ll use the Scikit-Learn library to build the feature vectors:


from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(min_df=5, max_features=5000)
review_tfidf = tfidf.fit_transform(preprocessed_reviews)


We now have a matrix review_tfidf where each row represents a review and each column is a word in the vocabulary weighted by its TF-IDF importance.

Sentiment Analysis Models

With our reviews converted into numerical feature vectors labeled by their 1-5 star ratings, we can train machine learning models to predict sentiment. Let‘s look at two popular approaches:

1. Logistic Regression

As a simple linear classifier, Logistic Regression learns a weighted sum of the input features and applies a sigmoid function to estimate the probability of the positive sentiment class. We can train it on our Flipkart data:


from sklearn.linear_model import LogisticRegression

sentiment_model = LogisticRegression()
sentiment_model.fit(review_tfidf, review_ratings)


2. LSTM Neural Network

For more advanced sentiment analysis, we can utilize deep learning sequence models like Long Short-Term Memory (LSTM) networks. LSTMs can process reviews as a sequence of word embeddings and identify long-range semantic dependencies. An example architecture looks like:


from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense

sentiment_model = Sequential()
sentiment_model.add(Embedding(vocab_size, 100, input_length=max_review_length))
sentiment_model.add(LSTM(64))
sentiment_model.add(Dense(1, activation=‘sigmoid‘))
sentiment_model.compile(loss=‘binary_crossentropy‘, optimizer=‘adam‘, metrics=[‘accuracy‘])
sentiment_model.fit(review_sequences, review_ratings, epochs=5, batch_size=64)


After training, we can apply our model to new reviews to predict their sentiment. A score closer to 1 indicates positive sentiment, while a score near 0 indicates negative sentiment.

Model Evaluation

To gauge how well our sentiment model generalizes to unseen data, it‘s important to evaluate on a separate test set. We‘ll split our labeled Flipkart dataset into train/test portions and compare the model‘s predicted sentiment scores to the true review ratings.

Some standard evaluation metrics for sentiment analysis include:

1. Accuracy

The percentage of reviews where the predicted sentiment matched the actual rating. Better suited for balanced datasets.

2. Precision/Recall

Precision measures what percentage of reviews predicted as positive were actually positive. Recall measures what percentage of truly positive reviews were correctly predicted as positive. There is often a tradeoff between optimizing for precision vs recall.

3. F1 Score

The harmonic mean of precision and recall, providing a balanced measure of a model‘s accuracy on both positive and negative classes.

4. ROC AUC

Plots the tradeoff between true positive rate and false positive rate. Models with better ROC AUC scores are more discriminative across different threshold settings.

Here‘s an example of evaluating our logistic regression sentiment model on a test set:


from sklearn.metrics import accuracy_score, f1_score

test_preds = sentiment_model.predict(test_tfidf)
print(‘Test Accuracy:‘, accuracy_score(test_ratings, test_preds))
print(‘Test F1 Score:‘, f1_score(test_ratings, test_preds))


Achieving high accuracy on the test set indicates that our model effectively learned meaningful patterns from the review text to predict customer sentiment.

Insights and Applications

Sentiment analysis on Flipkart reviews provides valuable insights for multiple stakeholders:

1. Flipkart

By aggregating sentiment scores across all reviews for a given product, Flipkart can identify which offerings are well-received vs poorly-received by customers. This can inform decisions to promote or discontinue certain products. Examining the text of negative reviews can also surface specific issues like late delivery, damaged packaging, inconsistent sizing, etc. Flipkart can use these insights to address problem areas and improve the customer experience.

Over time, Flipkart can also monitor sentiment trends to evaluate the impact of changes like a website redesign, new policies, or shifts in product strategy. If average sentiment scores are declining, that‘s an actionable signal to investigate and make corrections.

2. Product Manufacturers

Manufacturers can zoom in on reviews and sentiment for their specific products sold on Flipkart. If certain items are accumulating negative reviews, it may indicate design flaws or quality issues to address in future iterations. On the other hand, elements that are frequently praised in positive reviews can be emphasized in marketing and packaging.

User feedback can also inspire ideas for entirely new products to develop. If many reviewers are satisfied with a product but wish it had some additional feature, that could present an opportunity to introduce an upgraded version.

3. Customers

Shoppers on Flipkart can use sentiment analysis as a guide to make more informed purchase decisions. Sorting or filtering products by their average sentiment scores allows quickly identifying the best-rated options. This is more efficient than manually reading through hundreds of reviews.

Even when examining reviews for a single product, sentiment scores can facilitate quickly locating the most glowing positive reviews and the most scathing negative reviews. These polarized opinions often contain the most informative details to help evaluate a product‘s strengths and weaknesses.

4. Researchers

The Flipkart reviews dataset presents interesting opportunities for academic research into sentiment analysis techniques and linguistic trends. Researchers can experiment with different text representations, machine learning architectures, and training strategies to optimize model performance.

Analysis can also extend beyond just positive/negative classification to more fine-grained emotion detection – e.g. identifying reviews expressing anger, joy, surprise, disgust in addition to just overall sentiment. Drilling down into the language used in each emotional category could yield intriguing insights into how people express themselves in online platforms.

Conclusion

Sentiment analysis is a powerful tool for unlocking insights from the vast amounts of opinion data available on e-commerce platforms like Flipkart. By training machine learning models to classify the sentiment expressed in customer reviews, we can quickly distill thousands of text documents down into an overall indicator of customer satisfaction and detailed understanding of what elements are driving those opinions.

This empowers businesses to be more responsive to customer feedback, make data-driven decisions about their products and strategy, and optimize the user experience to maximize positive sentiment. Customers also benefit from easily evaluating the best-rated products that align with their needs.

As the scale of e-commerce and online reviews continues to grow exponentially, sentiment analysis will only become more essential for efficiently processing customer opinions. Especially when combined with other dimensions like helpfulness scores, user profiles, external social media data, there is immense potential to discover deeper and more nuanced insights.

However, realizing that potential requires careful design of machine learning pipelines to ensure high-quality data, robust models, and ethical application. Businesses and researchers must prioritize transparency in communicating the capabilities and limitations of sentiment analysis, and proactively mitigate risks around user privacy, demographic biases, and feedback loops.

Done properly, sentiment analysis creates a positive feedback cycle between businesses and customers. Flipkart‘s reviews provide a rich window into the customer experience – by intelligently analyzing that data, the company can make meaningful enhancements, shoppers can make better decisions, and the flywheel keeps turning.