A/B test evaluation
A focused review of a completed or running A/B test, so the team knows what the data supports before it ships or stops.
Book a diagnostic intro
Who it’s for
Results look mixed
The numbers point in more than one direction.
A win is in doubt
No one is sure the lift is real.
Decisions are waiting
Ship or stop hinges on a result no one trusts.
Fit
Well suited when
You have run, or are running, an A/B test on a live product, and you can share the raw data. You want a clear read on what it shows.
Poorly suited when
You have no live product or test yet. Planning your first experiments is a different service.
How it works
1
Brief
The change tested, the goal, the setup, and what the team expects the result to say.
2
Test data
Raw test and event data with table notes, or a person who knows them. First data request.
3
Statistical review
Tuesday and Thursday check-ins on sample quality, significance, and what the data can support.
4
Verdict readout
What the test shows, how sure we can be, the evidence, and what to do next.
5
Optional: next round
Continue together to design and run the next tests, scoped separately.
Your expected result gets tested too, not assumed. A clear winner is sometimes a measurement quirk.
What you receive
Setup and KPI review
Significance and sample check
Segment and effect breakdown
Ship, stop or iterate recommendation
90-day testing follow-up
If the test cannot support a conclusion, I will say so and explain what more is needed.
Examples
The test result was not what the team expected
Existing evidence argued against an executive’s proposal, so I turned it into a bounded live experiment rather than a permanent commitment. After designers addressed the risks, the test performed well and supported a wider release.
A test was misread because of how the data was counted
A shared-account test was read at member level even though the purchasing user was the relevant unit. Recounting around that user corrected the interpretation and clarified what the result supported.
Questions