Analyze A/B Test Results and Determine Statistical Significance
Get a complete A/B test analysis with statistical significance, confidence intervals, power analysis, and ship decisions.
📝 The Prompt
Act as an expert data analyst and statistician specializing in experimentation and A/B testing. I need you to perform a thorough analysis of the following A/B test and provide actionable recommendations.
**Experiment Details:**
- Test name/feature: [TEST_NAME, e.g., New checkout button color]
- Hypothesis: [HYPOTHESIS, e.g., Changing the CTA button from blue to green will increase conversion rate]
- Primary metric: [PRIMARY_METRIC, e.g., purchase conversion rate]
- Secondary metrics: [SECONDARY_METRICS, e.g., click-through rate, average order value, bounce rate]
- Test duration: [DURATION, e.g., 14 days]
- Traffic split: [SPLIT, e.g., 50/50]
**Observed Data:**
- Control group: [CONTROL_VISITORS] visitors, [CONTROL_CONVERSIONS] conversions
- Treatment group: [TREATMENT_VISITORS] visitors, [TREATMENT_CONVERSIONS] conversions
- Additional metric data (if available): [ADDITIONAL_DATA]
**Please deliver the following analysis:**
1. **Statistical Significance Test:** Calculate the p-value using an appropriate test (z-test or chi-squared). State whether the result is significant at the [SIGNIFICANCE_LEVEL, e.g., 95%] confidence level. Show your calculations step by step.
2. **Effect Size & Confidence Interval:** Compute the relative lift, absolute difference, and the confidence interval for the difference in conversion rates.
3. **Power Analysis:** Assess whether the sample size was sufficient to detect a meaningful difference. If not, calculate how many more samples would be needed.
4. **Segment Analysis:** Suggest 3-4 important segments to break down results by (e.g., device type, new vs. returning users) and explain what to look for.
5. **Validity Threats:** Identify potential issues such as novelty effect, selection bias, sample ratio mismatch, or peeking problems that could compromise results.
6. **Business Recommendation:** Based on the analysis, provide a clear ship/don't ship/extend test recommendation with reasoning.
Format the output with clear sections, include formulas used, and present a summary decision table at the end.
💡 Tips for Better Results
Plug in your actual numbers for visitors and conversions to get precise calculations rather than generic formulas.
Always include secondary metrics to catch cases where the primary metric improves but overall user experience degrades.
If your test ran for less than one full business cycle (typically 1-2 weeks), mention this so the AI can flag seasonality concerns.
🎯 Use Cases
Product managers, growth analysts, and data scientists should use this after completing an A/B test to rigorously evaluate results before making launch decisions.