Master new skills with our 21-day learning paths, broken into easy 5-minute daily lessons.

Start your journey for free.

martech Intermediate 21 lessons

A/B Testing Statistics

Run valid experiments. Master the statistics behind A/B testing: Sample sizes, Statistical Significance, Frequentist vs Bayesian methods.

Most A/B tests are invalid due to bad math. This course teaches the statistical rigour needed for experimentation. You will learn to calculate Sample Sizes to avoid underpowered tests. Understand P-values, Confidence Intervals, and the difference between Frequentist and Bayesian approaches. We cover common pitfalls like 'peeking' at results too early and Simpson's Paradox. Ensure your data-driven decisions are actually driven by valid data.

100% Free & Lifetime Access
⏱️ 5-Minute Lessons (Bite-sized learning)
🚀 21-Lesson Path (Independent modules)
📱 Mobile Friendly (Learn anywhere)
Experimentation
Start Learning
Secure Enrollment via SSL

Complete Course Syllabus

  • 1
    Stats Fundamentals
    Null Hypothesis, P-values, and Significance levels.
  • 2
    Sample Size
    Calculating how long to run a test for validity.
  • 3
    Bayesian vs Frequentist
    Different approaches to probability in testing.
  • 4
    Common Pitfalls
    Peeking, seasonality, and false positives.
  • 5
    Analysis
    Deciding when to roll out or roll back features.

Estimated completion time: 21 lessons • Self-paced learning • Lifetime access

Career Outlook

Estimated Salary
$100k - $140k

Career Paths

Experimentation Lead $110k-$150k
CRO Manager $90k-$130k
Product Analyst $95k-$135k

What You Will Learn

Design statistically valid A/B tests with correct sample sizes
Interpret P-values and Confidence Intervals accurately
Choose between Frequentist and Bayesian testing models
Identify and avoid common testing pitfalls like Peeking
Analyze test results to make confident business decisions

Skills You Will Gain

Statistics Experiment Design Hypothesis Testing Data Analysis Conversion Optimization

Who Is This For

CRO Specialists
Product Managers
Data Analysts

Prerequisites

Basic Math
Marketing concepts

A/B Testing Statistics FAQs

Math heavy?

Conceptual understanding is key; calculators do the math.

Tools?

We use generic calculators and industry platforms.

For devs?

Yes, helps implement feature flags correctly.

Why fail?

Most tests fail; learning *why* is the value.

Start Learning