ConversionXL (Georgi Georgiev) – Statistics for AB Testing

Price: $499

Become a proper optimizer and know your stats

If youre not fluent in A/B testing statistics, you wont be able to tell whether your tests suck.

A lot of your winning tests are probably not winners at all. Learn to call bullshit when needed, and be the person who advocates proper scientific approach in your team.

In 8sessions, youll learn

• How to run A/B tests with a sound statistical design.
• How to ask the right questions, avoid common mistakes, and get insights through statistics.
• An in-depth understanding of statistical hypothesis testing and concepts like statistical significance, confidence intervals, statistical power, and others.

Understand the complexities involved in planning and evaluating A/B tests

Avoid costly testing mistakes stemming from misuse and misunderstanding of statistics, and improve the ROI of all your A/B testing efforts, with Georgi Georgievs guidance.

This course is right for you if

• You cant define statistical significance correctly without looking it up on Google.
• Your A/B tests produce a lot of winners, but your clients arent seeing improvements.
• Youre planning and analyzing A/B tests, but you dont understand the statistical underpinnings of the testing process.
• Youre not confident in the outcomes of your tests and are unsure how much trust to put in them
• You have an in-house statistical tool you want to improve, or you use a third-party A/B testing software you want to understand better

This course is probably not for you if

• You are just starting with CRO and have little to no practical experience with A/B testing.
• You dont employ A/B tests as a primary method to evaluate CRO work.
• You are a professional statistician or experimental design specialist.

Skills you should have before taking this course:

• Some experience in conversion rate optimization.
• Basic understanding of how A/B testing works.
• Some experience with an A/B testing software.

About your instructor,
Georgi Georgiev

Hes the mastermind behind Analytics-Toolkit.com, a SaaS used by web analysts and CRO practitioners from agencies across the world.

Georgi Georgiev is the owner of WebFocus, a digital marketing consulting company delivering world-class marketing and analytics services for the past 10 years.

Georgi is a lecturer in multiple marketing events, as well as a Google Regional Trainer in AdWords & Analytics. He is also the author of three papers and multiple articles on A/B testing for conversion rate optimization.

In just 8 sessions, youll be able to

• Plan maximally efficient A/B tests.
• Correctly interpret A/B testing statistics.
• Navigate the complexities of MVT, segmentation, multiple KPIs, and concurrent tests.
• Plan and analyze sequential tests.

Your full course curriculum

Statistics for A/B testing

Lesson 1

Why A/B test? Basics of causal inference

In the first class, well lay the groundwork that’s required in order to understand more advanced concepts in subsequent classes. Well go over basic concepts that are crucial for developing a probabilistic mindset.

Topics covered:

• Correlation and causation. Observational analysis versus controlled experiments.
• Sampling and natural variance and their implications for drawing insights from data.
• Null-hypothesis statistical tests history and basics of causal inference.
• Control and randomization in A/B tests why we need them and how they work?
• One-sided and two-sided tests, composite vs. point hypothesis.

Lesson 2

Statistical significance & confidence intervals

Statistical significance is one of the most abused concepts in conversion rate optimization. You will learn what it is, really. Well discuss common misuses and misunderstandings, their consequences, and how to avoid them.

Topics covered:

• What is statistical significance.
• Common misuses of statistical significance and how to avoid them.
• Common misinterpretations and how to avoid them.
• A/A, A/B/A, A/A/B/B testing when are they appropriate, and what can they be used for?
• Confidence intervals.

Lesson 3

Planning A/B tests: Sample size & statistical power

Why is statistical power so important, and yet so neglected? You will learn about the trade-offs involved in planning A/B tests, and how to avoid under- and over-powered tests.

Topics covered:

• What is statistical power and why does it matter?
• The relationship between power and other statistical parameters: significance, sample size, & minimum detectable effect.
• Under-powered and over-powered tests.

Lesson 4

Multivariate testing & concurrent tests

Learn how to properly plan and analyze a multivariate test, avoiding common pitfalls. We examine the practice of running multiple concurrent tests, and how and when it’s appropriate.

Topics covered:

• Complexities introduced by testing more than one variant versus control
• When is an A/B/n test preferred to a simple A/B test?
• Dos and donts of running concurrent tests

Lesson 5

Segmentation, multiple KPIs, & non-binomial tests

Learn how to gain deeper insights by segmentation. Well also examine the fine details of using multiple outcome metrics for a test and cover non-binomial metrics such as revenue per user.

Topics covered:

• Segmenting A/B testing data for maximum insights.
• Complexities in running tests with more than one outcome metric.
• Analyzing non-binomial data such as revenue and time on site.

Lesson 6

Sequential testing

Sequential testing is the future of A/B testing. Learn about different approaches to sequential testing, and how to plan and analyze a sequential test.

Topics covered:

• Shortcomings of classical fixed-sample tests
• The issue of optional stopping
• The alpha-spending approach to sequential testing
• Planning and analyzing a sequential test
• Adaptive tests benefits and drawbacks

Lesson 7

Faster testing by asking the right questions

How can we run A/B tests with maximum efficiency? By designing and analyzing them to reflect the questions we want answered.

Topics covered:

• One vs two-tailed significance tests
• Non-inferiority testing
• Nontraditional hypothesis and analysis

Lesson 8

Planning ROI-positive A/B tests

The cherry on top: how to combine everything from the past seven courses to run highly efficient A/B tests that result in great returns.

Topics covered:

• Costs and benefits in A/B testing
• Planning ROI-positive A/B tests

Get ConversionXL (Georgi Georgiev) – Statistics for AB Testing immediately!