A Complete Guide to Market Basket Analysis in R
If you‘ve ever shopped online and seen recommendations like "Customers who bought this item also bought…", then you‘ve experienced the power of market basket analysis firsthand. Market basket analysis is a data mining technique that uncovers associations between items frequently purchased together by customers. For retailers, these insights are marketing gold – they can be used to optimize store layouts, offer relevant product recommendations, design effective promotions, and more.
In this article, we‘ll take a deep dive into market basket analysis using R. You‘ll learn the fundamental concepts, see detailed code examples, and get practical tips for uncovering actionable insights from your own retail data. Whether you‘re an aspiring data scientist or an experienced analyst, this guide will equip you with a valuable tool for your data science toolkit. Let‘s get started!
Understanding Market Basket Analysis
At its core, market basket analysis is all about finding interesting relationships between items in a store or on a website. The goal is to figure out which products customers often purchase together in the same transaction.
For example, a grocery store might find that customers who buy hamburger buns also tend to buy ground beef and ketchup in the same trip. Armed with this insight, the store could place those items close together on shelves, offer a promotion on burger fixings, or show shoppers personalized bundles based on items already in their cart.
To crunch the numbers, market basket analysis relies on a few key measures:
-
Support: The fraction of transactions that contain a particular item or itemset. It shows an item‘s overall popularity.
-
Confidence: The conditional probability of buying item B given that you purchased item A. It measures the likelihood that two items are bought together.
-
Lift: The ratio of the observed support to the expected support if the items were independent. A lift greater than 1 means items A and B are bought together more often than expected by random chance.
The most popular algorithm used in market basket analysis is Apriori. Apriori works by identifying frequent itemsets that meet a minimum support threshold, then generating association rules from those itemsets that also satisfy a minimum confidence threshold.
Here‘s a small example to illustrate. Let‘s say we have a dataset with 100 transactions:
- 10 transactions contain both milk and bread
- 8 transactions contain milk alone
- 15 transactions contain bread alone
The support for {milk, bread} is 10/100 = 10%. The confidence of the rule "if milk, then bread" is 10/18 = 56%. And the lift is (10/100) / ((18/100)*(25/100)) = 2.22.
A lift higher than 1 suggests that milk and bread are associated, meaning customers tend to buy them together. The store could use this rule to make sure milk and bread are stocked near each other.
Now that we‘ve covered the key concepts, let‘s see how to perform market basket analysis in R!
Preparing Your Data for Market Basket Analysis
Most often, transaction data comes in a format with one row per transaction and item names or IDs in columns, like this:
| Transaction ID | Item 1 | Item 2 | Item 3 |
|---|---|---|---|
| 1 | Milk | Bread | Eggs |
| 2 | Bread | Butter | |
| 3 | Milk | Yogurt | Bananas |
To make this data suitable for market basket analysis in R, we need to convert it to a "transactions" format. In this format, each row is a transaction, and each column represents an item. The cells show a 1 if the item was present in that transaction or 0 if not.
R makes this data transformation easy with the transactions() function from the arules package. Here‘s an example:
library(arules)
# dataframe with one row per transaction
trans_df <- data.frame(
TID = c(1,2,3),
Item1 = c("Milk","Bread","Milk"),
Item2 = c("Bread","Butter","Yogurt"),
Item3 = c("Eggs",NA,"Bananas")
)
trans <- transactions(trans_df)
print(trans)This gives us a transactions object that looks like:
items transactionID[2] {Bread,Butter} 2
[3] {Milk,Yogurt,Bananas} 3
Now our data is ready for association rule mining! Before we generate rules, it‘s often helpful to do some exploratory analysis on item frequencies first.
Exploring Item Frequencies
Before generating association rules, it‘s useful to get a feel for your data by looking at item frequencies. You can see which items are most common and rare, and start thinking about support thresholds to use later.
To create an item frequency bar plot in R:
library(arules)
data(Groceries) # load example Groceries dataset
itemFrequencyPlot(Groceries,topN=20,type="absolute")This code produces a plot showing the 20 most frequent items in the Groceries dataset:
We can see that whole milk, other vegetables, rolls/buns are among the most common items. With this context, we‘re ready to mine our data for interesting associations.
Mining Association Rules
The apriori() function from the arules package makes it simple to mine a transaction dataset for association rules. You just need to provide the minimum support and confidence thresholds.
Let‘s generate rules with at least 0.1% support and 50% confidence:
rules <- apriori(Groceries, parameter = list(supp = 0.001, conf = 0.5))
inspect(rules[1:5])lhs rhs support confidence lift [2] {pasta} => {whole milk} 0.0058 0.5049505 1.772668
[3] {herbs} => {root vegetables} 0.0037 0.5000000 6.780935
[4] {herbs} => {other vegetables} 0.0068 0.5000000 2.721981
[5] {beef} => {root vegetables} 0.0074 0.5033557 6.826989
This shows the top 5 rules found along with their support, confidence and lift. We can interpret the first rule as:
- Of all transactions, 0.33% contain both pot plants and whole milk
- 50% of customers who bought pot plants also bought whole milk
- Customers who bought pot plants were 1.76 times as likely to buy whole milk compared to the average customer
We can then use summary() to get an overview of the mined rules:
summary(rules)This returns useful info like the distribution of rule lengths, a summary of rule quality measures, and more.
Handling Redundant Rules
One issue you might notice is redundant rules. These are rules that don‘t provide any additional information because they‘re already captured by a more general rule.
For example, let‘s say we have these two rules:
- {yogurt} => {whole milk} (support=0.05, confidence=0.6)
- {yogurt, berries} => {whole milk} (support=0.01, confidence=0.6)
The second rule doesn‘t tell us anything new – the same proportion of people who buy yogurt, with or without berries, also buy whole milk. The berries are redundant.
To automatically prune redundant rules in R:
subset.matrix <- is.subset(rules, rules)
subset.matrix[lower.tri(subset.matrix, diag=T)] <- NA
redundant <- colSums(subset.matrix, na.rm=T) >= 1
rules.pruned <- rules[!redundant]Now we have a condensed set of non-redundant rules to explore further!
Targeting and Sorting Rules
Often, we‘re interested in finding rules for specific items, either on the left-hand-side or right-hand-side. For example:
- What items do people tend to buy before purchasing whole milk? (whole milk on RHS)
- What items are customers likely to buy in addition to whole milk? (whole milk on LHS)
To find rules with a specific item on the RHS, we can modify our apriori() function:
rules_milk_rhs <- apriori(data=Groceries, parameter=list(supp=0.001, conf=0.08),
appearance=list(default="lhs", rhs="whole milk"),
control=list(verbose=F)
)
rules_milk_rhs <- sort(rules_milk_rhs, decreasing=TRUE, by="confidence")
inspect(rules_milk_rhs[1:5])And to put whole milk on the LHS and see what customers buy with it:
rules_milk_lhs <- apriori(data=Groceries, parameter=list(supp=0.001, conf=0.15, minlen=2),
appearance=list(default="rhs", lhs="whole milk"),
control=list(verbose=F)
)
rules_milk_lhs <- sort(rules_milk_lhs, by="lift", decreasing=TRUE)
inspect(rules_milk_lhs[1:5])Notice that we set minlen=2 to avoid getting rules with whole milk alone on the LHS. We also sort by lift to surface the most interesting associations.
Visualizing Association Rules
A picture is worth a thousand rules! The arulesViz package offers several ways to visualize association rules to help convey insights.
One useful plot is a network graph, which shows rules as nodes connected by arrows. Bigger nodes represent higher support items. Thicker arrows indicate higher confidence rules.
library(arulesViz)
plot(rules, method="graph",Engine="interactive", shading="lift")
Another handy visualization is a grouped matrix plot. It displays the lift measure for rules with a certain item on the LHS vs RHS. Darker squares mean higher lift.
plot(rules, method="grouped", Engine="interactive")
These interactive plots make it easy to spot interesting rules and communicate results to stakeholders.
Choosing Support and Confidence Thresholds
You may have noticed that the apriori() function requires setting minimum support and confidence thresholds. But how do you choose these values? Too high and you might miss important rules. Too low and you‘ll get too many low-value results.
Here are a few tips:
-
Start with a support threshold around 0.1% to 1% depending on how many unique items and transactions you have. Higher support for smaller datasets.
-
Choose a high confidence like 80% to focus on the strongest associations. Lower to 50% or less if you want to allow weaker but potentially novel findings.
-
Adjust support and confidence based on the business context. Be stricter if you‘re automatically generating recommendations to avoid low-quality results. Be more lenient if you‘re simply exploring the data.
-
Use the
summary()andinspect()functions to assess rule quality and rerun with different thresholds if needed. Look at the distribution of support, confidence and lift. -
Consider setting a maximum rule length to avoid overly specific rules that are harder to act on. 3-4 items per rule is often a good target.
The key is to experiment and iterate! Market basket analysis is as much an art as a science.
Advanced Topics and Resources
We‘ve covered the essentials of market basket analysis in R, but there‘s always more to learn. Here are a few advanced topics worth exploring:
- Using market basket analysis for product recommendations, either manually or automated
- Performing market basket analysis on web clickstreams and ecommerce data
- Optimizing store layouts and product placement based on association rules
- Extending the Apriori algorithm to handle other metrics besides support, confidence, lift
- Trying alternative algorithms like FP-Growth and comparing performance vs Apriori
- Dealing with very large sparse datasets using the
arulespackage and efficient data formats
Here are some helpful resources to dive deeper:
- Introduction to Market Basket Analysis in R
- A gentle introduction on Market Basket Analysis — Association Rules
- Complete Guide to Association Rules (1/2)
- Market Basket Analysis at Instacart
- arules Reference Manual
I also recommend browsing real-world transaction datasets to practice your market basket analysis skills. The Instacart Market Basket Analysis dataset on Kaggle is a great place to start.
Conclusion
Phew, that was a whirlwind tour of market basket analysis in R! We covered a lot of ground, from the basic concepts to detailed code examples to advanced tips and tricks. I hope you now feel equipped to uncover valuable insights from your own retail data.
To recap, the key steps for market basket analysis are:
- Prepare your transaction data in a suitable format
- Explore item frequencies to understand your data
- Mine association rules using the Apriori algorithm
- Prune redundant rules
- Target specific items and sort/filter rules to find the most interesting ones
- Visualize your rules to communicate insights
- Iterate with different support and confidence thresholds
Market basket analysis is a powerful tool for any retailer or ecommerce business. By understanding which products customers frequently buy together, you can optimize everything from store layouts to product bundles to personalized recommendations. With the arules package, R makes it easy to perform market basket analysis on your own data.
So what are you waiting for? Fire up RStudio and start uncovering hidden gems in your transaction data. Your customers (and your bottom line) will thank you. Happy mining!
