Results rose: was it our change?
Why before-and-after numbers are not enough, and what helps assess the impact of a change.
- ROSSAUS · 011 / 5
Results rose after a change
Conversion rose from 5 % to 8 %. That is a useful signal, but the timing alone does not explain the cause.
5 → 8 %conversion after the changeA teaching example about test impact - ROSSAUS · 022 / 5
What else changed around it?
Seasonality, a discount or audience mix may have contributed. I record those changes and look for a comparable period or a control group.
- SeasonDifferent demand
- PriceA discount or promotion
- AudienceA different traffic mix
Separate the product change from external factors - ROSSAUS · 033 / 5
Add a group without the change
In a suitable A/B test, groups are randomly assigned and compared over the same period. Here, the test gained 3 percentage points and the control gained 1: a 2-point difference in changes.
+2 ppdifference in changes- Test5 → 8 %
- Control5 → 6 %
A teaching example, not proof of impact without checking the data - ROSSAUS · 044 / 5
Check the comparison quality
I review data volume and quality, group comparability, the observation window and uncertainty. Sparse data do not support a confident conclusion.
- 01Enough data
- 02Consistent measurement rules
- 03No harm to later steps
The main metric should not improve at the expense of enquiry quality or retention - ROSSAUS · 055 / 5
A decision with its evidence
I record the change, the comparison and what we learned. Then we decide whether to extend the test, implement the change or stop. Data guide the next step.
Read the next breakdowns- ChangeOne hypothesis
- MeasureOutcome and limits
- DecideThe next action
I will keep unpacking decisions like these in my notes
Use the buttons or ← → keysSwipe or use the buttons