A/B Testing

What A/B Testing Is

A/B testing (also called split testing) splits traffic randomly between two versions of something, a landing page, an email subject line, a pricing table, an onboarding flow, and measures which version wins on a metric you define ahead of time, like signup rate or click-through rate. Version A is usually the current experience (the control) and version B is the change you're testing (the variant). Whichever version produces a statistically significant lift gets rolled out to everyone.

The core idea is isolating one variable at a time so you know the outcome was caused by your change, not by chance, seasonality, or who happened to visit that day.

Why It Matters for Early-Stage Founders

Early on, you have almost no data and every decision (headline, CTA copy, pricing, onboarding steps) is a guess. A/B testing replaces opinion-driven debates ("I think the blue button converts better") with evidence. It's also one of the few growth tools available before you have a large team or big budget: you don't need new features, just traffic and a hypothesis.

That said, A/B testing has a floor. If you get 50 visitors a week, you'll wait months for a valid result. Early-stage founders should mostly use A/B testing for high-traffic, high-leverage pages (your homepage, signup flow, pricing page) and use qualitative methods (user interviews, session recordings) everywhere else until traffic grows.

How It Works: The Basic Steps

1. Pick one metric that matters (e.g., signup conversion rate, not "engagement" broadly).
2. Form a hypothesis: "Changing the CTA from 'Get Started' to 'Start Free Trial' will increase signups because it removes ambiguity about cost."
3. Build variant B and randomly split traffic 50/50 between A and B.
4. Run the test until you hit your required sample size, not just "until it looks like a winner."
5. Check for statistical significance before declaring a winner.
6. Ship the winner (or, if flat, learn and move to the next test).

Calculating Sample Size and Significance

The two numbers founders most often skip are minimum sample size and confidence level, which is how most "winning" tests turn out to be noise.

Simple formula for required sample size per variant:

n ≈ 16 * p * (1 - p) / d²

Where p is your baseline conversion rate and d is the minimum detectable effect (the smallest lift you actually care about), both as decimals.

Example: Your signup page converts at 5% (p = 0.05) and you want to detect a lift to 6% (a 1 percentage point, or 20% relative, improvement, so d = 0.01).

n ≈ 16 * 0.05 * 0.95 / (0.01)² ≈ 7,600 visitors per variant

That means roughly 15,200 total visitors needed before you can trust the result, at a standard 95% confidence level. If your site gets 2,000 visitors a week, that's nearly two months just to detect a modest lift. This is why founders with low traffic should test bigger, bolder changes (a full page redesign, not a button color) that might produce a 2x lift instead of a 20% one, since bigger effects need smaller samples.

A Worked Example

Suppose you A/B test two homepage headlines over three weeks:

B shows a 20% relative lift. Before rolling it out, run these numbers through a significance calculator (many are free online). At this sample size, a jump from 5.0% to 6.0% typically clears the 95% confidence threshold, meaning there's less than a 5% chance the difference is due to random variation. Only then do you ship B to 100% of traffic.

Common Mistakes

• Peeking and stopping early. Checking results daily and stopping the moment B "pulls ahead" inflates false positives dramatically. Decide your sample size in advance and stick to it.
• Testing too many variables at once. Changing headline, image, and CTA simultaneously means you won't know what actually drove the result. That's multivariate testing, a different (and more data-hungry) method.
• Ignoring statistical significance entirely. A 5.2% vs 5.4% conversion rate on 300 visitors is noise, not a winner.
• Testing low-traffic pages. A pricing page with 40 visitors a month will never reach significance in a reasonable timeframe. Prioritize your highest-traffic funnel steps.
• Optimizing a metric that doesn't matter. A headline that boosts clicks but not paid signups isn't a real win. Always tie the test metric to a business outcome.

Benchmarks and Rules of Thumb

• Run tests for at least one full business cycle (usually 1 to 2 weeks minimum) to smooth out day-of-week effects.
• Aim for at least a few hundred conversions per variant before trusting results; thousands is safer for small effect sizes.
• Use 95% confidence as the standard bar before shipping a change; some teams use 90% for low-risk, easily-reversible tests.
• If you're pre-product-market-fit with limited traffic, prioritize qualitative feedback and big directional bets over formal A/B tests, and save rigorous split testing for after you have consistent traffic, something teams often revisit around launch time, which is also where a checklist from a resource like welaunch.sh can help sequence what to test first.