luna/marketingskills
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: address codex review feedback on growth experiments

Browse Source 
Soften win-rate language to context-aware guidance. Clarify weekly
cadence to monitor guardrails while avoiding premature winner calls.
Expand playbook template with sample size, CI, guardrails, segment
deltas, and implementation status.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Corey Haines
2026-04-01 15:46:03 -07:00
parent 7c13fccdd9
commit c4cb25cb6c
1 changed files with 8 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/ab-test-setup/SKILL.md
+8 -4
View File
@@ -279,7 +279,7 @@ Track your experimentation rate as a leading indicator of growth:
| Metric | Target | | Metric | Target |
|--------|--------| |--------|--------|
| Experiments launched per month | 4-8 for most teams | | Experiments launched per month | 4-8 for most teams |
| Win rate | 20-30% is healthy (if higher, you're not being bold enough) | | Win rate | 20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses) |
| Average test duration | 2-4 weeks | | Average test duration | 2-4 weeks |
| Backlog depth | 20+ hypotheses queued | | Backlog depth | 20+ hypotheses queued |
| Cumulative lift | Compound gains from all winners | | Cumulative lift | Compound gains from all winners |
@@ -292,17 +292,21 @@ When a test wins, don't just implement it — document the pattern:
## [Experiment Name] ## [Experiment Name]
**Date**: [date] **Date**: [date]
**Hypothesis**: [the hypothesis] **Hypothesis**: [the hypothesis]
**Result**: [winner/loser] — [primary metric] improved by [X%] (p=[value]) **Sample size**: [n per variant]
**Why it worked**: [analysis of why] **Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"] **Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work] **Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]
``` ```
Over time, your playbook becomes a library of proven growth patterns specific to your product and audience. Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.
### Experiment Cadence ### Experiment Cadence
**Weekly (30 min)**: Review running experiments for technical issues. Don't analyze results — just check they're collecting data correctly. **Weekly (30 min)**: Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.
**Bi-weekly**: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog. **Bi-weekly**: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.