Analytics & Measurement
A/B testing
Also known as: split testing, A/B test, AB testing
A/B testing is a controlled experiment that shows two or more versions of a page, email or feature to randomly split groups of users and compares how each performs against a chosen goal. It isolates the effect of a single change so improvement can be attributed rather than assumed. It is also called split testing.
What it is
In an A/B test, visitors are randomly assigned to a control version and one or more variants, with everything else held constant. Results are compared on a primary metric such as conversion rate, then assessed for statistical confidence before a winner is declared. Multivariate testing extends the idea by testing combinations of several elements at once.
Why it matters
Opinions about what works on a page are cheap and often wrong, and seasonal or traffic mix changes can make simple before and after comparisons misleading. Randomised testing gives a defensible answer about whether a change helped, hurt or did nothing, which protects budget and stops teams shipping changes that quietly reduce revenue. It also builds a record of what your specific audience responds to.
How it works
Teams define a hypothesis and a primary metric, calculate the sample size needed to detect a meaningful difference, then run the test until that sample is reached rather than stopping when results look good. Testing tools handle the random split and reporting, while server side testing is used where flicker, speed or personalisation matter. Results are reviewed alongside secondary metrics and segment breakdowns to check the winner is not helping one group and harming another.
When it applies
Use it when you have enough traffic and conversions to detect a realistic effect size in a sensible timeframe, and when the change is reversible. For low traffic pages or one-way decisions, qualitative research, expert review and sequential measurement are usually more practical.
Examples
- An online retailer tests a single product image against a short looping video on a category page, measuring add to basket rate.
- A SaaS company tests a free trial call to action against a book a demo call to action on the same homepage hero.
- A charity tests two donation form layouts, one with suggested amounts and one with an open field, measuring completed donations.
How it is measured
- Primary conversion rate difference between control and variant, with confidence interval
- Revenue or value per visitor, to catch cases where conversions rise but order value falls
- Sample size reached and test duration against the pre-agreed plan
- Win rate and average lift across the testing programme over a quarter
Insights on A/B testing
Related terms in Analytics & Measurement
- AttributionAttribution is the practice of assigning credit for a conversion to the marketing touchpoints that preceded it. It covers the models, rules and data joins used to decide which channels, campaigns or content get counted. In AI search, attribution is harder because many assistant-led journeys leave little or no referral data.
- Click-through rateClick-through rate is the proportion of people who saw something and then clicked it, expressed as clicks divided by impressions. It is used to judge search listings, ads, emails and internal links. A higher rate usually means the message matched what the audience was looking for.
- Consent ModeConsent Mode is a Google framework that lets tags adjust their behaviour based on the consent choices a visitor has made. Instead of tags being blocked outright, they receive signals about whether analytics and advertising storage are allowed, and act accordingly. Version 2 added parameters covering the use of personal data for ads and for ad personalisation.
- Conversion trackingConversion tracking is the practice of recording the actions you care about, such as purchases, form submissions or calls, and connecting them back to the channel, campaign or session that led to them. It gives advertising platforms and analytics tools the outcome data they need to report performance and optimise bidding. Accuracy depends on correct tag implementation, consent handling and clear conversion definitions.
- Cross-channel reportingCross-channel reporting is the practice of bringing performance data from search, social, email, paid media, AI assistants and other channels into a single view. It standardises metrics and time periods so channels can be compared fairly rather than judged in isolated platform dashboards. The aim is to show how channels work together to produce enquiries, sales and revenue.
- Customer lifetime valueCustomer lifetime value (CLV or LTV) is the total profit or revenue a business expects to earn from a customer relationship over its lifetime. It combines purchase value, purchase frequency, margin, and retention into a single forward looking figure. Teams use it to decide how much they can afford to spend acquiring and keeping customers.
Primary research · August 2026
How ChatGPT Shortlists Software Brands
An audit across 10 categories and 60 buying questions. I recorded what ChatGPT reads, throws away and links to when a buyer asks it which software to buy, and what that decides.
- 60
- Questions asked
- 10
- Software markets
- 2,680
- Results read
- 367
- Links shown
Free download
Get the full report
35 pages · PDF · 536 KB. Enter your details and it downloads straight away.