Lessons from The Seer‘s Accuracy ML Competition: Do Seers Need to Be Accurate?

Introduction

Machine learning (ML) competitions have become a popular way for data scientists to hone their skills, learn from their peers, and see how their modeling approaches stack up. Typically, these competitions provide a structured dataset with a predefined prediction target, and challenge entrants to develop models with the best possible accuracy or other metric.

However, the recent Seer‘s Accuracy competition hosted by Analytics Vidhya took a different approach. Participants were given only a raw dataset, without a specified target variable or train/test split. The challenge was to first formulate the business problem and prediction task themselves, before building the most accurate model possible.

This twist tested skills beyond just model building, such as problem framing, feature engineering, and validation approaches. Over 2,200 data scientists took part in the 3-day hackathon, vying to predict which customers of the fictional electronics retailer ElecMart would make repeat purchases.

In this post, we‘ll dive deep into the winning solutions from this unique competition and extract lessons that can be applied to real-world ML projects. We‘ll also step back and consider the role of accuracy in ML more broadly – how important is it for machine learning "seers" to have high accuracy, versus other characteristics like interpretability or ease of implementation? Let‘s explore.

Deconstructing the Winning Approaches

While the Seer‘s Accuracy dataset didn‘t have a predefined target variable, most participants realized the core task was to predict the probability of a customer making another purchase in a future period (2006), based on their 2003-2005 transaction history. However, the top solutions went far beyond this basic problem setup in their feature engineering and modeling approaches.

Feature Engineering

The 3rd place finisher, Bishwarup Bhattacharjee, engineered an extensive set of features for each customer, including:

• Demographic info: Age of the customer as of 2007-01-01
• Categorical associations: Customer to store, product category, sales rep, lead source, and payment method (one-hot encoded into sparse matrices)
• Temporal aggregations: Counts of transactions in last 1/3/6/12 month windows
• Transaction amount metrics: Min, max, mean, range, median
• Timing signals: Days since last purchase, time to first purchase
• Ratios: Unique stores visited / total purchases, unique product categories / total purchases

The 2nd place team of Oleksii Renov and Thakur Raj Anand constructed similar features, with a few notable twists:

• They created 4 different data representations, hypothesizing each would capture different predictive signals
• For each data representation, they built both linear and tree-based models, and found that ensembling the outputs provided a boost over either model type alone (more details on this in the Modeling section below)

Rohan Rao, the competition winner, also invested heavily in feature engineering. His feature set was similar in breadth to the other winners, but meticulously validated to avoid leakage or target contamination. He credited his win to:

"Really drilling down to understand the problem, building a robust validation scheme, and ensembling 3 XGBoost models trained on different subsets of the data. My best single model was able to get within 1% of my winning submission‘s AUC score."

It‘s clear that extensive feature engineering was key to squeezing the best performance out of this dataset. As data scientist Luca Massaron writes:

"Feature engineering is the art of extracting useful patterns from raw data…done right, it can make all the difference between a weak, inaccurate model and an effective one that generates real business value."

Modeling and Validation

While their feature engineering approaches had some unique elements, all of the winning solutions relied heavily on the popular XGBoost algorithm for modeling. XGBoost‘s ability to handle sparse data representations and its robust performance across many domains made it a natural choice for this tabular dataset.

Some key XGBoost modeling insights shared by the winners:

• Oleksii Renov found that blending the outputs of XGBoost and linear models (logistic regression) yielded an appreciable boost in AUC compared to using either model type alone, as shown in Table 1 below. This reflects the classical Yin-Yang ensembling approach.

• Rohan Rao‘s winning solution was an ensemble of 3 different XGBoost models, each trained on different subsets of the data and with different hyperparameters. He found this ensembling smoothed out the variance of any individual model. His best solo model had an AUC of 0.870, while his ensemble reached 0.880.

Table 1. Impact of blending XGBoost and linear model predictions, from Oleksii Renov

All the winners also emphasized the importance of a robust validation scheme when working with time series data like this. Simple train/test splits can lead to leakage and overfit models. Instead, they used techniques like:

• Time-based splitting (training on 2003-2005 data to predict 2006 reorders)
• Group K-fold cross validation (grouping all of a customer‘s data in the same fold)
• Leave-one-out validation at the customer level

As one winner put it: "Always trust your offline CV – and to trust your CV, you need to build the right validation method for the problem at hand."

The Role of Accuracy in Machine Learning

The Seer‘s Accuracy competition, like most ML contests, ranked solutions based on a single metric – AUC in this case. The winning solutions achieved impressive AUC scores between 0.87-0.88, indicating highly accurate predictions of customer repeat purchases.

However, this raises an interesting question – do machine learning models, and their human "seer" counterparts, truly need such high levels of accuracy to deliver real-world value? Is the ML community too focused on measures like AUC at the expense of other important considerations?

The answer likely depends on the use case. In some applications, like high-frequency trading or medical diagnosis, even small improvements in accuracy can be incredibly valuable (and errors incredibly costly). A 2021 study found that increasing the AUC of a cancer screening model by just 0.05 could save hundreds of lives per year.

For a retailer like the fictional ElecMart, highly accurate predictions of which customers will repurchase could enable more efficiency in targeting retention efforts and promotional spend. Wasting marketing on customers who would have reordered anyway, or failing to entice those on the fence, cuts directly into profits.

However, there are certainly business scenarios where approximate or directional predictions are good enough. Consider an e-commerce company using an ML model to personalize product recommendations on their website. While recommending the single most likely next purchase for a customer might be ideal, showing products with a 30-40% chance of purchase can still meaningfully improve the customer experience and drive incremental sales. Perfect accuracy isn‘t essential.

The same logic applies for many demand forecasting use cases. Predicting that sales will grow by roughly 10% next quarter can help with resource planning, even if the actual number comes in a few points higher or lower. The prediction doesn‘t need to be exact to deliver value.

As legendary statistician George Box famously said, "All models are wrong, but some are useful." The most accurate model isn‘t always the most useful one for a given situation.

This is especially true when accuracy comes at the cost of other desirable model characteristics like:

• Interpretability: Can stakeholders understand how the model works and why it‘s making certain predictions? This is important for building trust and diagnosing issues.
• Ease of implementation: How much engineering effort is required to productionize the model and keep it running reliably? A slightly less accurate model that‘s easy to implement may be preferred.
• Computational efficiency: Can the model generate timely predictions given system constraints? A 1% boost in accuracy isn‘t worth much if predictions take 10x longer.
• Maintenance: How much work is needed to retrain the model over time and keep it current? High-effort models can quickly become stale.

In academic settings like ML competitions, these practical tradeoffs are often abstracted away. But in industry, they‘re critical considerations. A complex, opaque model that tops the leaderboard may be less useful than a simple, transparent one that‘s easier to work with.

As data scientist Cassie Kozyrkov puts it:

"Always start by considering the context of the problem you‘re solving, your goals, and the constraints you‘re working with. Then pick a performance metric well-aligned with delivering value under those circumstances."

The Limits of Accuracy

It‘s also important to recognize the fundamental limits of predictive accuracy, given the inherent uncertainty and complexity of the real world. No model will ever be perfect, and unforeseen events can always disrupt even the most precise forecasts.

In some cases, chasing accuracy above all else can actually be counterproductive. If a model is overfit to past data, it may break when future data inevitably differs. Highly complex "black box" models can also be brittle and prone to unexpected behavior.

There are also risks of getting so caught up in model tuning that we overlook issues like:

• Biased or incomplete training data
• Data leakage or contamination
• Feedback loops between predictions and outcomes
• Non-stationarity in the system we‘re modeling

No amount of hyperparameter optimization can fix a flawed dataset or problem setup. Accuracy on a historical validation set also doesn‘t guarantee sustained real-world performance over time.

So while accuracy can be a useful guidepost, especially for well-scoped problems like the Seer‘s challenge, it shouldn‘t be worshipped uncritically. Responsible AI/ML practitioners must always maintain a healthy skepticism about their models, and consider accuracy in the broader context of the system they‘re embedded in.

Conclusion

The Seer‘s Accuracy competition showcased how thoughtful problem framing, feature engineering, and model ensembling can extract highly accurate predictions even from a poorly structured dataset. The winning solutions provide a masterclass for aspiring ML competitors.

At the same time, the competition sparked reflections on the role and importance of accuracy in applied ML. Pushing the boundaries of predictive performance is incredibly valuable for advancing the field. But accuracy is not the be-all end-all, and must be balanced against other priorities like interpretability, practicality, and generalization.

Ultimately, the goal should be to develop ML models that are useful, not just accurate – and "usefulness" must be defined in the context of each specific application. For some use cases, solo models with middling accuracy will suffice. For others, ensembles of highly tuned models like the Seer‘s competition winners are appropriate.

As more and more organizations adopt machine learning, it‘s critical that practitioners remain grounded in the real-world impacts of their work. We must strive to be responsible "seers" – aiming for accuracy while staying cognizant of its limits and unintended consequences. Only then can we realize the full potential of AI/ML to help make better decisions and drive positive outcomes.