A/B Testing Measurement Frameworks Every Data Scientist Should Know

As a data scientist, A/B testing should be a key part of your experimentation toolkit. A/B testing, also known as split testing, is a method of comparing two versions of a web page, app, or other digital experience to determine which one performs better. By randomly splitting traffic between the two versions, you can use statistical analysis to determine which version has a higher conversion rate or achieves another desired outcome.

A/B testing takes the guesswork out of website optimization and enables data-informed decision making. For companies looking to optimize conversions, user engagement, or other critical metrics, A/B testing is essential. And as a data scientist, you play a central role in the A/B testing process by determining metrics, implementing the test, analyzing the results, and drawing data-driven conclusions.

In this post, we‘ll dive into the key components of a successful A/B testing framework that every data scientist should know. By the end, you‘ll understand the different types of metrics to track, how to determine sample size and duration, common pitfalls to avoid, and a step-by-step process you can follow. Let‘s get started!

Key Components of an A/B Testing Framework

A well-designed A/B testing framework includes a few key components:

• Metrics
• Statistical concepts
• Sample size
• Duration

Let‘s break each of these down further.

Metrics

One of the most important parts of an A/B test is deciding what you will measure. There are a few main types of metrics to consider:

Macro metrics: These are the overarching metrics that impact your business‘ bottom line, like revenue, conversion rate, or customer lifetime value. The goal of your A/B test should be to improve a macro metric.

Guardrail metrics: These are metrics you want to make sure don‘t get negatively impacted by your test. For example, if you‘re testing a new landing page, a guardrail metric could be bounce rate. You want to make sure the new page doesn‘t cause a significant increase in bounce rate.

Micro metrics: These are more granular metrics that can help explain movements in your macro metrics. Micro metrics are often leading indicators of macro metrics. For an e-commerce site, micro metrics could include add-to-cart rate, email signup rate, or average order value.

When designing your A/B test, choose one macro metric that aligns with your business goals as your primary metric. Then select a handful of guardrail and micro metrics as secondary metrics to help you interpret the test results.

Here are some examples of metrics for different types of A/B tests:

• E-commerce: conversion rate, average order value, revenue per visitor
• Lead generation: form completion rate, email signup rate, lead quality score
• Media/publishing: pageviews per session, bounce rate, ad revenue per session
• SaaS: free trial signups, paid conversions, first-month retention

Statistical Concepts

To determine whether an A/B test result is statistically significant, you need a basic understanding of a few key statistical concepts:

Hypothesis testing: An A/B test is set up to test a hypothesis, such as "the new version will have a higher conversion rate than the old version."

Null hypothesis: The null hypothesis states that there is no difference between the two versions. We assume the null hypothesis is true until we have sufficient evidence to prove otherwise.

Statistical significance: Statistical significance is the likelihood that the difference in conversion rates between the control and treatment versions is not due to random chance. It‘s typically expressed as a p-value between 0 and 1. A p-value less than 0.05 is considered statistically significant by most standards.

Statistical power: Statistical power is the probability that a test will detect a real difference between the versions when one exists. The higher the power, the more likely you are to correctly reject the null hypothesis and avoid a false negative. 80% power is a common threshold.

Type I error: A Type I error, also known as a false positive, is when you incorrectly reject the null hypothesis. The p-value represents your Type I error rate.

Type II error: A Type II error, also known as a false negative, is when you fail to reject the null hypothesis when you should have. The beta value represents your Type II error rate.

These concepts get quite math-heavy, but luckily there are many A/B testing calculators (like this one) that will do the calculations for you. Still, it‘s important to understand the basic principles so you can interpret the outputs correctly.

Determining Sample Size

Before launching an A/B test, you need to calculate the minimum sample size required to detect a significant difference between the control and treatment versions. The formula for calculating sample size relies on a few inputs:
– Baseline conversion rate
– Minimum detectable effect (the smallest improvement in conversion rate you want to be able to detect)
– Significance threshold (typically 95%)
– Statistical power (typically 80%)

The higher your baseline conversion rate and minimum detectable effect, the smaller the sample size required. Choosing a minimum detectable effect requires striking a balance – an effect that is too small will require a very large sample size and a lot of time to reach significance. But an effect that is too large may cause you to miss out on small but meaningful lifts.

You can calculate sample size using an online calculator or by using statistical software. Remember that the sample size output is per variation, so you‘ll need to collect that number of samples for both your control and treatment versions.

Determining Test Duration

There are two main approaches to determining how long to run an A/B test for: fixed horizon and sequential.

In a fixed horizon test, you choose a sample size and test duration up front based on your estimated conversion rate and minimum detectable effect. Once you reach the predetermined sample size, you stop the test and analyze the results. The benefit of this approach is that it‘s simple to implement and avoids the risk of peeking at the data and ending the test prematurely. The downside is that if your estimates are off, you may end up collecting more samples than you need to reach significance.

In a sequential test, you continuously monitor the results and stop the test as soon as you reach a significant result. This approach is more statistically efficient, but it‘s also riskier since you may introduce bias by choosing to stop when you see a significant result by chance.

In general, a fixed horizon test is the better choice for most A/B tests. Plan to run the test for full weeks at a time (to account for weekday/weekend differences) until you reach your calculated sample size. For most tests, 1-2 weeks is sufficient.

A Step-by-Step A/B Testing Process

Now that we‘ve covered the key concepts, let‘s walk through the process of running an A/B test from start to finish.

1. Decide on your hypothesis and primary metric. What do you think the new version will improve, and how will you measure that?

2. Choose your secondary metrics. What other metrics do you want to keep an eye on to make sure the new version doesn‘t have unintended consequences?

3. Calculate your sample size. Use a sample size calculator to determine how many visitors you‘ll need in each version to detect your desired effect.

4. Implement the test. Use an A/B testing tool like Optimizely or Google Optimize to set up your test and split traffic between the versions.

5. Monitor the results. Keep an eye on your primary metric and make sure your guardrail metrics aren‘t being negatively impacted. Avoid the temptation to end the test early if you see significant results.

6. Analyze and draw conclusions. Once you‘ve reached your predetermined sample size, stop the test and analyze the results. If the change is statistically significant, roll it out to all users. If not, use your secondary metrics to help explain why.

7. Plan your next test!

A/B Testing Case Studies

Let‘s take a look at a couple real-world examples of A/B testing in action.

E-Commerce Checkout Flow Test

An e-commerce company wanted to increase its conversion rate by improving the checkout flow. They hypothesized that reducing the number of form fields required would increase conversion rate without decreasing average order value.

Primary metric: conversion rate
Secondary metrics: average order value, form field abandonment rate, revenue per visitor

They calculated that a 5% lift in conversion rate from a baseline of 2% would require a sample size of about 8,000 visitors per version. So they ran a test for 2 weeks until they collected slightly more than 8,000 samples in each version.

The results showed an 8% lift in conversion rate that was statistically significant at the 95% level (p-value < 0.05). Average order value and revenue per visitor also increased, while form field abandonment decreased. Based on these positive results, the team decided to roll out the new checkout flow to all users.

SaaS Pricing Page Test

A SaaS company tested three different pricing tiers on its pricing page: basic, professional, and enterprise. They hypothesized that adding a fourth "premium" tier at a higher price point would increase revenue by shifting more users into the professional and enterprise tiers.

Primary metric: average revenue per user
Secondary metrics: distribution of signups across pricing tiers, trial-to-paid conversion rate

They used A/B testing software to split traffic between the original 3-tier version (control) and the new 4-tier version (treatment). After one month, they had enough samples to detect a significant difference in average revenue per user, which had increased in the treatment version by 12% (p-value < 0.01).

Looking at the secondary metrics, they saw that the treatment version had a higher percentage of signups at the professional and enterprise tiers compared to the control. The trial-to-paid conversion rate was not significantly different between the two versions. Based on these results, they launched the 4-tier pricing structure to all users.

Common A/B Testing Pitfalls to Avoid

We‘ve covered a lot of best practices for A/B testing, but there are also some common mistakes to watch out for:

Stopping a test too early: It can be tempting to stop a test as soon as you see significant results, but this increases your chances of a false positive. Decide on a sample size before the test and stick to it.

Running underpowered tests: On the flip side, make sure your sample size is sufficiently large to detect a meaningful effect. Use a sample size calculator for each test.

Changing your metrics mid-test: Decide which metrics you will use to evaluate the test before you launch. Changing metrics or introducing new variations mid-test can invalidate the results.

Ignoring seasonality: Be mindful of how seasonality may impact your test, especially if you‘re running it over a major holiday. Try to run tests for full weeks at a time to account for day-of-week fluctuations.

Not following up: The learnings from an A/B test are only valuable if you take action on them. Plan out your next steps for rolling out the winning variation or investigating why a test didn‘t show significant results.

Conclusion

A/B testing is a powerful tool for data scientists and businesses to optimize digital experiences and products. By comparing a control and treatment version and measuring the difference across key metrics, you can determine whether a change will have a positive impact.

To run effective A/B tests, data scientists need to understand how to select primary and secondary metrics, calculate sample size and duration, and analyze the results. Avoiding common pitfalls like underpowered tests, changing metrics mid-flight, and ending tests too early will help ensure your results are reliable.

There is always more to learn about A/B testing statistics and methodology. I recommend the following resources for further reading:

• A Refresher on A/B Testing (Harvard Business Review)
• Guidelines for A/B Testing (Hooked on Data)
• A/B Testing (Optimizely)
• Google Analytics A/B Testing Course

No matter what type of business you work for, experimentation and A/B testing should be a key part of your data science workflow. By creating a rigorous A/B testing framework, you‘ll be able to measure the impact of changes, make data-informed decisions, and drive long-term business value. Happy testing!