Building a Content-Based Product Recommendation System

The way people shop has undergone a dramatic transformation over the past couple decades. Back in the early 2000s, most people still purchased products by visiting physical stores. Fast forward to today, and e-commerce has exploded in popularity, with millions of people buying everything from groceries to furniture to electronics through online websites and apps. Global e-commerce sales surpassed $5 trillion in 2022 and are projected to continue growing rapidly in the coming years.

The COVID-19 pandemic accelerated the shift to online shopping, as lockdown restrictions prevented many physical stores from operating normally. Even as the pandemic subsides, the convenience of browsing massive product catalogs and ordering items for home delivery with a few taps on a screen is likely to sustain the growth of e-commerce. By 2025, e-commerce is expected to account for nearly a quarter of total global retail sales.

The Crucial Role of Recommendation Systems

In this intensely competitive e-commerce landscape, engaging customers effectively is crucial for retailers to thrive. With so many products to choose from across countless online stores, capturing a shopper‘s attention and keeping them coming back to your site or app is hugely challenging. This is where recommendation systems come into play.

By leveraging data on customer preferences and behavior, e-commerce recommendation engines suggest products that a shopper is likely to be interested in. This helps customers discover relevant products they may not have otherwise come across, while driving incremental sales for the retailer. When done well, recommendations create a virtuous cycle – users are exposed to products they love, so they purchase more and remain loyal, providing the retailer with more data to further refine their recommendations.

Some of the most prominent tech companies have demonstrated the massive impact of recommendation systems:

• 35% of Amazon‘s revenue is generated by its product recommendations
• 75% of what people watch on Netflix comes from recommendations rather than search
• 70% of YouTube viewing time is driven by its AI-powered recommendation engine

Types of Recommendation Systems

There are three main approaches for building recommendation systems:

1. Popularity-based: Recommends products that are universally popular across all users, e.g. a news site displaying the current top trending articles. Simple to implement but not personalized.

2. Content-based: Recommends products that are similar to the ones a user has engaged with in the past, based on product attributes like category, brand, description, etc. Requires detailed product metadata but no data on other users.

3. Collaborative filtering: Identifies users with similar preferences and recommends products they have liked to each other, e.g. "viewers who watched this movie also liked…". Powerful when you have a lot of user interaction data but suffers from the "cold start" problem for new users and items with no ratings yet.

4. Hybrid: Combines two or more of the above approaches to offset their individual shortcomings. A common technique is to use content-based filtering to address the cold start problem for new products, then switch to collaborative filtering once the item has some interactions.

For the rest of this post, we‘ll dive deep into content-based filtering and walk through building a product recommendation engine using this approach.

Understanding the Dataset

To demonstrate how to build a content-based recommender, we‘ll use a public dataset from Kaggle containing product listings from Bigbasket, a popular online grocery delivery service in India. The dataset has nearly 20,000 unique products across almost 2,000 brands in categories like beverages, beauty, and household supplies. For each product, we have metadata like:

• product name
• category and sub-category
• brand
• type
• item description
• price
• average user rating

Let‘s start by reading the data into a Pandas DataFrame and checking the shape:

import pandas as pd

df = pd.read_csv(‘Bigbasket_Products.csv‘) 
print(df.shape)

(19190, 10)

We have 19,190 products and 10 attributes, which is a decent sized dataset to build our recommendation engine. Next, let‘s check for missing values:

print(df.isnull().sum())  

index             0
product         115
category          0
sub_category      0
brand             0
sale_price        0
market_price      0
type            115
rating          115
description     115
dtype: int64

There are 115 products with missing values for the name, type, rating, and description. Since these attributes are crucial for our content-based filtering, let‘s remove these products:

df = df.dropna()
print(df.shape)

(19075, 10)

After dropping missing values, we still have over 19,000 products remaining.

Exploratory Data Analysis

Before jumping into building the recommendation system, it‘s important to understand the distribution of values across the key attributes we‘ll be using. Let‘s start with the "category" column:

df[‘category‘].value_counts().head(10).plot(kind=‘bar‘)

The "Beauty & Hygiene" category contains by far the most products with over 7000 items, followed by "Kitchen, Garden & Pets" and "Household & Cleaning" with around 2000-2500 each. Understanding where the bulk of the product catalog lies is important context.

Next, let‘s look at the distribution of user ratings:

df[‘rating‘].hist(bins=20)  

The vast majority of products have a rating of 4.0 or higher out of 5.0. This positively skewed distribution could reflect actual high product quality or rating system biases. We should be somewhat cautious in relying too heavily on ratings in our model.

Finally, let‘s engineer a new feature called "discount_percent" that captures the % difference between the market price and sale price for each product:

df[‘discount_percent‘] = (df[‘market_price‘] - df[‘sale_price‘]) / df[‘market_price‘] 

print(df[‘discount_percent‘].describe())   

    count    19075.000000
    mean         0.075165
    std          0.141581
    min          0.000000
    25%          0.000000
    50%          0.000000
    75%          0.090909
    max          0.933333
    Name: discount_percent, dtype: float64

The mean discount is 7.5% but the median is 0%, meaning most items are not discounted at all. However, the max discount is over 93% so there are a few steeply discounted products. Having a sense for the range and skew of numeric features is helpful for evaluating if they need any normalization.

Building the Content-Based Recommender

Now that we‘re familiar with the key characteristics of our product catalog, let‘s get into actually building the recommendation engine. The core idea behind content-based filtering is that if a user expressed interest in a particular item in the past, they are likely to also engage with products that have similar attributes.

So our goal is to represent each product in our catalog as a multi-dimensional vector capturing all its metadata attributes, then calculate a similarity score between the vectors. When a user interacts with a specific product, we‘ll recommend other products that have the highest similarity scores.

Step 1: Combine Metadata Attributes

First, we need to combine the relevant product metadata columns into a single feature that we‘ll vectorize. Let‘s use the category, sub-category, brand, and type columns:

def combine_features(row):  
    return row[‘category‘] + ‘ ‘ + row[‘sub_category‘] + ‘ ‘ + row[‘brand‘] + ‘ ‘ + row[‘type‘]

df[‘metadata_combined‘] = df.apply(combine_features, axis=1)  
df[‘metadata_combined‘].head(3)  

0    Beauty & Hygiene Hair Care Streax Hair Colour...
1          Household & Cleaning Air Fresheners Odonil
2    Beauty & Hygiene Skin Care Nivea Body Lotions...
Name: metadata_combined, dtype: object

Step 2: Vectorize the Combined Metadata

Next, we need to turn the combined metadata string for each product into a numerical vector. We‘ll use the bag-of-words model and a TF-IDF vectorizer from scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words=‘english‘)  
tfidf_matrix = tfidf.fit_transform(df[‘metadata_combined‘])  

The TF-IDF vectorizer converts our combined metadata into a matrix where each row represents a product and each column corresponds to a word from the metadata. The cell values capture the importance of each word to the product.

Step 3: Calculate Similarity Scores

To determine how similar two products are, we calculate the cosine similarity between their vectorized metadata. Cosine similarity measures the angle between two vectors – a score of 1 means the items are identical while a score of -1 means they are exact opposites.

from sklearn.metrics.pairwise import cosine_similarity

cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)  

We can now look up the similarity scores between any two products in our catalog:

idx = df.index[df[‘product‘] == ‘Dove Body Love Body Lotion‘].tolist()[0]
print(idx)  

1298

cosine_sim[idx][df.index[df[‘product‘] == ‘Dove Nourishing Secrets Body Lotion‘]]  

array([0.69387698])

The Dove "Body Love" and "Nourishing Secrets" body lotions have a cosine similarity of 0.69, indicating they are quite similar as expected since they are from the same brand and product type.

Step 4: Get Top N Recommendations

Finally, we‘ll write a function that takes in a product name and returns the top N most similar products based on cosine similarity:

def get_recommendations(product, n=5):
    idx = df.index[df[‘product‘] == product].tolist()[0]
    similarities = list(enumerate(cosine_sim[idx]))
    top_similar = sorted(similarities, key=lambda x: x[1], reverse=True)[1:n+1]

    top_products = []
    for i in top_similar:
        top_products.append(df.iloc[i[0]].product)

    return top_products

Let‘s test it out:

get_recommendations(‘Magnum Ice Cream - Almond‘)  

[‘Kwality Wall‘S Ice Cream - Magnum Truffle‘,
 ‘Magnum Ice Cream Stick - Classic‘,
 ‘Ice Cream - Magnum Brownie‘,
 ‘London Dairy Ice Cream - Chocolate‘,
 ‘Kwality Walls Ice Cream - Chocolatey Conettes‘] 

The recommender identifies other chocolate/almond ice cream products from Magnum and competing brands as being the most similar, which seems quite relevant.

Refining the Model

While the initial recommendations look decent, let‘s see if we can improve them further. One additional signal we could incorporate is the actual product name itself, since things like flavor, key ingredients, and brand are often mentioned there.

To do this, we‘ll create a separate vectorizer for the product name, combine it with the metadata vectorizer, then recalculate the cosine similarities:

tfidf_name = TfidfVectorizer(stop_words=‘english‘)
tfidf_name_matrix = tfidf_name.fit_transform(df[‘product‘])  

combined_matrix = sparse.hstack([tfidf_name_matrix, tfidf_matrix])
cosine_sim = cosine_similarity(combined_matrix, combined_matrix)  

Now let‘s regenerate our earlier recommendations:

get_recommendations(‘Magnum Ice Cream - Almond‘)

[‘Magnum Ice Cream - Almond & White Chocolate‘,
 ‘Magnum Ice Cream Dessert Chocolate Truffles - Hazelnut‘,  
 ‘Magnum Dessert Chocolate Truffles - Almond‘,
 ‘Magnum Ice Cream Stick - Classic‘,  
 ‘Kwality Wall‘S Ice Cream - Magnum Truffle‘]

The new top recommendations seem more relevant, with a focus on other almond and white chocolate Magnum ice creams. The specific mention of "Almond" in the product name allows us to place higher importance on this flavor.

Conclusion

In this post, we walked through the key steps for building a content-based recommendation system:

1. Combine relevant item metadata attributes into a single feature
2. Vectorize the combined feature using a TF-IDF vectorizer
3. Calculate the cosine similarity between all item vectors
4. Retrieve the top N most similar items for a given product

We also saw how incorporating the product name into the vectorizer can improve the relevance of the recommendations.

Some of the key advantages of content-based filtering are:

• No need for user interaction data, so can make recommendations for new items
• Recommendations are highly relevant to the specific item
• Relatively straightforward to implement

However, there are a few significant limitations:

• Requires detailed product metadata, which may not always be available
• Recommendations are not personalized to each user
• Cannot recommend products from different categories than what the user has engaged with before

Therefore, content-based filtering is most effective when you have rich metadata for the items in your catalog but limited user behavior/interaction data. It‘s a great way to get a recommendation engine up and running quickly or to complement other approaches like collaborative filtering.

The natural next step to further improve our recommender would be to incorporate user purchase/interaction signals as they become available. We could use content-based similarity scores to address the cold start problem for new products, then gradually blend in collaborative filtering as items accumulate more interactions. A hybrid approach combining the strengths of different techniques is ultimately the most powerful.

I hope this guide gave you a solid foundation for building a content-based recommendation system. Feel free to connect with me on LinkedIn if you have any other questions!