A/B Test

Category: Product & PMF · Level: Entry · Also called: AB test, Split test, A/B testing

TL;DR

A controlled experiment that compares two versions of a feature, page, or flow to determine which produces a better outcome.

An A/B test randomly splits users into a control group (A) and a variant group (B), exposes each to a different experience, and measures the impact on a chosen metric. With enough sample size and a clean random assignment, the difference between the two groups can be attributed to the variant.

Doing it rigorously requires pre-registering the hypothesis, picking a primary metric, computing the required sample size in advance, and not peeking at the result early. Most teams underpower their tests and over-trust early signals.

Worked example

A pricing-page A/B test routes 50% of visitors to a $49/mo price and 50% to $79/mo. After 4 weeks and 8,400 visitors per arm, the $79 variant converts 1.4% vs 1.7% for $49 — but ARPU is 36% higher, so the $79 variant ships with 95% statistical confidence on revenue per visitor.

Common pitfalls

  • Calling a winner before reaching statistical significance.
  • Running too many simultaneous tests on the same surface and corrupting attribution.
  • Optimizing local UI changes while ignoring the larger funnel.

When this shows up in a pitch deck

Rarely a slide on its own, but A/B test discipline is implied when growth metrics improve substantially over a known baseline.

Related terms

  • Funnel Analysis — Decomposing a user journey into ordered steps and measuring conversion between each step to find the biggest drop-off.
  • Conversion Rate — The percentage of users who complete a desired action — sign-up, purchase, upgrade — out of those who had the chance to.
  • Activation Rate — The percentage of new sign-ups who reach the product's defined aha moment within a target time window.
  • Lean Startup — A methodology for building startups under uncertainty using rapid Build-Measure-Learn cycles instead of long product plans.
  • Cohort Analysis — Grouping users by sign-up period and tracking each group's behavior over time to spot trends invisible in aggregate metrics.

Use this in your next pitch deck

Deckmetric scores your pitch across 10 VC frameworks and against 8 investor types. Upload your deck for an instant analysis, or check the startup valuation calculator to benchmark your raise.