Statistical Significance Calculator

Result

Enter the numbers for both variants to test the difference for significance.

What is statistical significance, and what 95 % confidence promises

Statistical significance answers one question: is the difference between two variants real, or is it chance? When variant B shows a higher conversion rate than A, that can be a genuine improvement — or simply a handful of visitors who happened to buy that week. A significance test puts a number on how often the measured gap would appear if the two variants were in truth identical. Only once that number is small enough has the test earned the right to decide anything.

The p value formula, and the z score it comes from

This calculator runs a pooled two proportion z test. The two conversion rates p₁ and p₂ and the pooled rate p̂ produce a z score, and the standard normal distribution turns that z score into a two-sided p value:

p̂ = (c₁ + c₂) ÷ (n₁ + n₂)

z = (p₂ − p₁) ÷ √( p̂(1−p̂) × (1/n₁ + 1/n₂) )

The larger |z| grows, the harder the gap is to explain away as luck. The thresholds are |z| > 1.645 (90 %), |z| > 1.960 (95 %) and |z| > 2.576 (99 %). The p value says the same thing as a probability, which is the form worth taking into a meeting.

How to calculate statistical significance from visitors and conversions

Control A: 10,000 visitors, 200 conversions. Variant B: 10,000 visitors, 250 conversions:

At p ≈ 0.017 the value sits below 0.05, so the difference is significant at the 95 % level. Raise the bar to 99 % confidence, where the threshold is z > 2.576, and z = 2.38 is not enough — the same data stops counting. Nothing about the experiment changed, only how much doubt you were willing to live with.

One tailed vs two tailed: the setting that halves your p value

This calculator is fixed to a two-sided test. The two most visible A/B calculators on the market let you switch to one-sided, and the switch is not cosmetic: on identical data a one-sided test reports half the p value. The worked example above gives p ≈ 0.017 two-sided and would give ≈ 0.009 one-sided — significant at 99 % rather than merely at 95 %, from the same 10,000 visitors. One-sided is only defensible if, before you collected a single visitor, you were genuinely willing to ignore the possibility that the variant is worse. Almost nobody is; the variant that tanks checkout is exactly the result you needed to see. So if a test clears the bar here, it clears it anywhere. If it only clears someone else's one-sided setting, the honest reading is that the test is not finished.

The false positive rate you are accepting, and two other honest warnings

• Do not stop early («peeking»): Checking each morning and stopping the moment it looks significant pushes the false positive rate far above the 5 % you thought you chose. Fix the sample size and the runtime in advance, then leave both alone.
• Significant is not the same as important: Very large samples make even tiny, commercially pointless differences significant, and very small ones miss real effects completely. Read the uplift next to the p value, never instead of it.
• 95 % confidence is not 100 %: On average 1 test in 20 comes back significant when there is no real difference at all — that is the false positive rate you agreed to when you picked 95 %. A result you intend to build on is worth repeating.

The sample size formula belongs before the test, not after it

• Plan the sample first: The block above solves n = (1.96 + 0.84)² × (p₁(1−p₁) + p₂(1−p₂)) ÷ (p₂ − p₁)² for each variant. Run it before the test; run it afterwards and it only explains why the result was inconclusive.
• Run whole weeks: End a test on the same weekday it started. Weekday and weekend traffic convert differently, and a test cut short on a Tuesday carries that imbalance straight into the result.
• One change per test: Change several things at once and a significant result tells you that something worked — not which one. That is a finding you cannot reuse.