5 Winning Feature Engineering Strategies from the AmExpert Retail Hackathon

Introduction

In the dynamic world of retail, leveraging data science and machine learning has become critical for companies to stay competitive and meet evolving customer needs. One area where these techniques can have a significant impact is in coupon marketing campaigns. By accurately predicting which customers are most likely to redeem coupons, retailers can optimize their strategies, boost sales, and drive customer loyalty.

In 2019, American Express and Analytics Vidhya joined forces to host the AmExpert hackathon, challenging data scientists around the globe to develop innovative solutions for coupon redemption prediction. The hackathon provided a massive dataset from a real-world retail scenario, spanning customer demographics, transaction history, campaign details, and more.

Over 1000 data enthusiasts participated in the 9-day event, submitting nearly 15,000 solutions. The competition was fierce, but in the end, five exceptional teams rose to the top of the leaderboard. In this article, we‘ll dive deep into their winning approaches, with a special focus on the advanced feature engineering techniques that set them apart.

But before we get started, let‘s take a moment to understand the problem statement and dataset behind the AmExpert hackathon.

Problem Statement & Dataset Overview

The hackathon revolved around helping the fictional retailer ABC improve their coupon marketing process. The objective was to predict the probability of a customer redeeming a coupon, based on data from 18 previous campaigns. By building an accurate predictive model, ABC hoped to design more effective coupon constructs and targeted promotions.

The dataset provided was extensive, including:

• User demographic details
• Campaign and coupon specifics
• Product information
• 2 years of transaction data

Participants had to grapple with multiple data tables, including campaign data, coupon-item mappings, customer demographics, item details, and a large transaction table. The train set covered 18 campaigns, while the test set included the subsequent 10 campaigns for which predictions were required.

With over 5900 registered participants, the hackathon was hotly contested. Submissions were evaluated using the AUC-ROC metric between the predicted probabilities and actual redemption rates. After an intense battle, the top 5 entries achieved an impressive AUC score of over 0.94.

Now that we have the context, let‘s uncover the secrets behind their success, starting with rank 5.

Rank 5: Mudit Tiwari

Mudit‘s approach heavily focused on meticulous feature engineering, extracting insights from every table to build a comprehensive feature set. He created four distinct derived datasets by aggregating the granular transaction data in various ways to capture customer, item, and coupon-level behaviors.

One of Mudit‘s key realizations was the importance of the customer transaction table. He performed extensive aggregations on this data, taking care to avoid data leakage by only using transactions up to the start date of each campaign. His features encompassed various aspects like total discounts, percentage savings, and more.

Interestingly, Mudit observed that coupon-item mappings held valuable information. For each campaign and coupon combination, he looked at the customer‘s historical purchases of the applicable items. This item-level aggregation proved to be one of the most predictive features in his model.

Validation was another crucial aspect of Mudit‘s approach. After experimenting with different strategies like group K-fold, he settled on a traditional K-fold split which yielded better results. His final ensemble consisted of CatBoost, LightGBM, and XGBoost models, stacked using both logistic regression and a neural network.

Rank 4: Pradeep Pathak

Like Mudit, Pradeep‘s success was rooted in constructing an extensive feature set – 124 to be precise. He created aggregations at various levels, including customer-date, coupon-date, customer-coupon-date, and even brand-date levels.

Pradeep‘s key insight was to leverage historical transactions to generate features, but with careful subsetting to prevent data leakage. For each data point, he only considered transactions prior to the campaign start date. Daily-level aggregations for customer and coupon combinations also proved effective.

The final submission was an ensemble of LightGBM, XGBoost, and CatBoost models. Pradeep emphasized the importance of a reliable cross-validation strategy and leak-free feature engineering in securing a top spot on the leaderboard.

Rank 3: Sahil Verma

Sahil‘s approach shared similarities with the previous two, focusing heavily on creating informative features while avoiding leakage. He derived customer-level features indicating their propensity to purchase products related to each coupon based on historical transactions. Similar features were constructed at the brand and category levels.

Apart from buying propensities, Sahil calculated metrics like total spend, discounts availed, and more. He extended this idea to coupon-level features as well, estimating the probability of a coupon being redeemed based on its associated items‘ past performance.

Sahil‘s final model was a blend of CatBoost, LightGBM, and XGBoost, trained on two datasets differing in feature counts. His key takeaways were the importance of thoroughly understanding the problem statement, crafting leak-free features, and building a robust validation strategy.

Rank 2: Mohsin Khan

Mohsin‘s approach stood out for its unique problem formulation. He observed that the task could be viewed as ranking the coupons for each customer based on their likelihood of redemption. This ranking could be inferred from the customer‘s historical transactions – if a coupon contained items frequently bought by the customer, it would rank higher.

To make the problem computationally feasible, Mohsin employed resampling techniques to balance the dataset. He also ranked customers based on their overall propensity to redeem coupons, derived from the training data.

By framing the problem as a ranking task and cleverly leveraging the transactional data, Mohsin was able to build a highly effective solution that secured him the second place.

Rank 1: Sourabh Jha

Sourabh‘s winning approach was a masterclass in feature engineering. He constructed over 300 features, capturing intricate patterns at customer and coupon levels. His feature set included aggregations, latent embeddings, and time-weighted metrics.

Using Singular Value Decomposition (SVD), Sourabh created latent vectors for customers and coupons from various matrices like customer-item, customer-brand, and customer-coupon interactions. He also derived category and brand-wise metrics such as total spend, savings, and transaction counts.

To ensure the robustness of his model, Sourabh employed a 5-fold cross-validation strategy with random splitting. He trained LightGBM models on the engineered features, using permutation importance to identify the most predictive ones. This allowed him to reduce the feature set from 300+ to a more manageable 107.

Sourabh‘s key insight was that careful feature engineering, combined with leak-free validation, was vital to achieving a winning score. His use of latent embeddings to capture higher-level patterns was particularly innovative.

Actionable Insights for 2024 & Beyond

While the AmExpert hackathon took place in 2019, the learnings from the winning solutions remain highly relevant for retail data science in 2024 and beyond. Here are some key takeaways that practitioners can implement in their own work:

1. Feature engineering is king: Across all winning solutions, the common denominator was an emphasis on creating informative, domain-relevant features. Whether it‘s aggregations, latent embeddings, or time-weighted metrics, the quality of features can make or break a model‘s performance.

2. Avoid data leakage at all costs: When dealing with temporal data like transactions, it‘s crucial to ensure that features are constructed using only information available prior to the prediction point. Leakage can lead to overly optimistic cross-validation scores and poor real-world performance.

3. Experiment with problem formulation: As Mohsin‘s approach demonstrated, sometimes reframing the problem can lead to novel and effective solutions. Don‘t be afraid to look at the task from different angles and experiment with unconventional approaches.

4. Ensemble for the win: All top solutions leveraged ensembles of multiple models, typically combining gradient boosting algorithms like XGBoost, LightGBM, and CatBoost. Ensembling can help improve robustness and generalization performance.

5. Validate, validate, validate: A reliable cross-validation strategy is essential for assessing model performance and preventing overfitting. As seen in the winning approaches, techniques like k-fold, group k-fold, and careful data splitting are key to building trust in your models.

As we move further into the 2020s, the retail landscape will continue to evolve, driven by advancements in data science and shifting consumer preferences. However, the fundamental principles exemplified by the AmExpert winners – a deep understanding of the problem domain, creative feature engineering, and rigorous validation – will remain essential for success.

By learning from these winning approaches and adapting them to new challenges, data scientists can continue to drive innovation and deliver value in the dynamic world of retail. As customer expectations rise and competition intensifies, those who can harness the power of data to make accurate predictions and optimize decisions will be well-positioned to thrive.

Conclusion

The AmExpert hackathon may have taken place a few years ago, but its lessons are timeless. The winning solutions showcased the immense potential of feature engineering and machine learning in tackling real-world retail challenges like coupon redemption prediction.

By diving deep into the top five approaches, we‘ve seen how careful data preprocessing, creative feature construction, and robust validation can lead to remarkable results. From Mudit‘s item-level aggregations to Sourabh‘s latent embeddings, each winner brought a unique perspective and skillset to the table.

As aspiring data scientists and machine learning practitioners, we can learn a great deal from these experts. Their success stories underscore the importance of domain expertise, technical proficiency, and a relentless focus on solving the problem at hand.

In the rapidly evolving retail landscape of 2024 and beyond, these lessons will be more valuable than ever. As businesses seek to personalize customer experiences, optimize marketing spend, and drive growth, the ability to extract insights from vast troves of data will be a key differentiator.

So let‘s take inspiration from the AmExpert winners and continue pushing the boundaries of what‘s possible with data science in retail. By staying curious, experimenting relentlessly, and always keeping the end-user in mind, we can unlock new opportunities and drive meaningful impact in this exciting field.

Here‘s to the future of data-driven retail – may the insights be ever in your favor!