In this guide
Your next steps
- Choose one question and a primary outcome.
- Check comparable delivery and mature data.
- Read the effect size together with uncertainty.
- Record a decision and a follow-up test.
A better number is not yet a winning test
One ad has a higher CTR or lower purchase CPA. Before stopping the other, check whether the difference comes from the creative or from audience selection, timing and a few unsettled purchases.
A randomized experiment and a comparison of two existing ads are different forms of evidence. A dashboard can organize historical results; it cannot create random assignment after delivery.
1. Define the question before reading the result
Write a specific hypothesis: replacing the opening with a product demonstration should improve purchase rate for this audience. Keep the offer, landing page and other major elements stable if you want to isolate the opening.
Choose the primary metric, smallest worthwhile improvement, duration, sample requirement and stopping rule in advance. For a sales decision, clicks are a diagnostic rather than a substitute for purchases.
In scale190, open Comparisons → 1-on-1 tests → Create test to start a saved comparison of imported ads. This does not launch a randomized Meta advertising experiment.

Comparisons → 1-on-1 tests → Create test. Supplied screenshot with sample data; the arrow marks the starting point for a saved comparison.
Open full-size screenshot2. Check maturity, volume and delivery
Use the same observation window and allow for conversion delay. Check for promotions, stock changes, tracking gaps and site outages. A partial first day is not equivalent to a complete trading week.
Inspect both arms: a 90/10 delivery split leaves one with much less information. Required sample size depends on baseline rate, effect size, error tolerance and power; no universal impression threshold proves reliability. Repeated exposures and overlapping audiences also limit independence.
3. Separate the observed lift from certainty
B is ahead in this hypothetical example. The gap is 0.2 percentage points, or 20% relative lift. That arithmetic does not establish why B is ahead.
Read an appropriate interval or test alongside the practical effect. A frequentist p-value is not the probability that B is better. An inconclusive result does not prove equality; it may be underpowered.
| Measure | Ad A | Ad B |
|---|---|---|
| Impressions | 20,000 | 20,000 |
| Clicks (same definition) | 200 | 240 |
| CTR | 1.0% | 1.2% |
| Observed B lift | — | +0.2 pp / +20% relative |
4. Avoid manufacturing a winner
Do not repeatedly check a fixed-horizon test and stop when it first looks significant. Use the planned stopping rule or a method designed for sequential monitoring. Do not switch the primary metric after seeing the result.
Many variants and metrics create more opportunities for chance findings. Mark exploratory results clearly, account for multiple testing in formal claims, and confirm important findings with fresh evidence.
Review the evidence in scale190
The 1v1 comparison workflow brings imported ads, creative differences and performance evidence together. Readiness, delivery and data-quality checks accompany deterministic calculations for supported proportion metrics.
A comparison confidence score is not a universal probability of future success. Aggregate cost and revenue totals do not supply every variance input for CPA or ROAS significance tests. scale190 reads data; it does not create randomized Meta experiments or change delivery.
Finish with the result, limitations and next action. A confounded comparison can inform the next test without becoming a permanent creative rule.
Sources & further reading
Worked examples are hypothetical. References provide further methodological background.
- Meta Blueprint: designing A/B tests, creative tests and lift studiesMeasurement study types and interpretation training.
- Stata: power analysis for two-sample proportionsSample-size planning, effect sizes and allocation ratios.
- NIST: tests and confidence intervalsBackground on frequentist interpretation.
