Sales & marketing · Calculator companion · By K Imports
A/B conversion significance: practical guide
This guide explains how to use the Count.ie a/b conversion significance. Compare two conversion rates using a two-sided pooled proportions test. Follow the inputs, method and worked example below, then compare your own scenario.
Guide written .
Open the A/B conversion significance calculatorWhat to prepare
- Define the event, conversion, customer and attribution period before collecting data.
- Use spend and outcomes from the same reporting window.
- Distinguish revenue, contribution profit, recurring revenue and customer lifetime assumptions.
Understand the inputs
The calculation method
z = conversion-rate difference ÷ pooled standard error. The two-tailed p-value uses the normal distribution.
Worked example
Illustrative inputs and their result.
Example inputs
- Group A visitors
- 1000
- Group A conversions
- 100
- Group B visitors
- 1000
- Group B conversions
- 130
- Confidence threshold (%)
- 95
Calculated example result
- Relative Lift Percent
- 30%
- P Value
- 0.0355
- Statistically Significant
- Yes
How to interpret the result
A marketing metric depends on its attribution and cost definitions. A favourable ratio is not proof that advertising caused every sale or that future customers will behave the same way.
Compare campaigns with the same attribution window and outcome definition. Test a lower conversion rate or shorter retention period before extrapolating a promising result.
Common mistakes to avoid
- Do not double-count a customer or conversion reported by multiple channels.
- Revenue is not the same as profit available to fund acquisition.
- Small samples and changing cohorts can produce unstable comparisons.
Assumptions and sources
Requires random independent groups and at least 10 conversions and non-conversions per group. Plan the stopping rule before testing.
If your case falls outside this scope, use a more suitable calculator or contact us about an unclear method.