A/B Test Analyzer — is that result real, or is it noise?

1

Your experiment

What did you measure? Counts of successes out of totals. Confidence level How much evidence before you call it. Times you checked results Be honest — this changes the answer.

Control (A)

Visitors Conversions

Variant (B)

Visitors Conversions

Try a scenario:

2

Verdict

3

What the numbers look like

Confidence intervals the plausible range for each arm's true rate — overlap means you cannot separate them

Posterior belief about each rate Bayesian view — how strongly the data pin down each arm's true rate

4

How big should the next test be?

The most common reason an experiment is inconclusive is that it was never large enough to detect the effect you were hoping for. Plan for that up front.

Baseline rate percent, e.g. 5 for 5% Smallest lift worth detecting percent, relative Target power

?

What the numbers mean

p-value: If A and B were truly identical, this is the chance of seeing a gap at least this large by luck alone. It is not the probability that B is better.
Confidence interval: The range of true effects consistent with your data. If it contains zero, "no difference" is still on the table.
Statistical power: The chance your test would have detected the effect it found, had that effect been real. Below 80%, a null result mostly tells you the test was too small.
P(B > A): The Bayesian answer to the question people actually ask. Unlike a p-value, this is the probability that B's true rate exceeds A's.
Peeking: Checking a fixed-horizon test repeatedly and stopping when it looks good inflates your false-positive rate far above 5%. Decide the sample size first, then look once.