fix: address codex review feedback on growth experiments

Soften win-rate language to context-aware guidance. Clarify weekly cadence to monitor guardrails while avoiding premature winner calls. Expand playbook template with sample size, CI, guardrails, segment deltas, and implementation status. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: add growth experimentation program to ab-test-setup
2026-04-01 15:46:03 -07:00 · 2026-04-01 15:15:18 -07:00
2 changed files with 96 additions and 57 deletions
@@ -1,8 +1,8 @@
 ---
 name: ab-test-setup
-description: When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," or "how long should I run this test." Use this whenever someone is comparing two approaches and wants to measure which performs better. For tracking implementation, see analytics-tracking. For page-level conversion optimization, see page-cro.
+description: When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program," or "experiment playbook." Use this whenever someone is comparing two approaches and wants to measure which performs better, or when they want to build a systematic experimentation practice. For tracking implementation, see analytics-tracking. For page-level conversion optimization, see page-cro.
 metadata:
-  version: 1.1.0
+  version: 1.2.0
 ---

 # A/B Test Setup
@@ -229,6 +229,93 @@ Document every test with:

 ---

+## Growth Experimentation Program
+
+Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.
+
+### The Experiment Loop
+
+```
+1. Generate hypotheses (from data, research, competitors, customer feedback)
+2. Prioritize with ICE scoring
+3. Design and run the test
+4. Analyze results with statistical rigor
+5. Promote winners to a playbook
+6. Generate new hypotheses from learnings
+→ Repeat
+```
+
+### Hypothesis Generation
+
+Feed your experiment backlog from multiple sources:
+
+| Source | What to Look For |
+|--------|-----------------|
+| Analytics | Drop-off points, low-converting pages, underperforming segments |
+| Customer research | Pain points, confusion, unmet expectations |
+| Competitor analysis | Features, messaging, or UX patterns they use that you don't |
+| Support tickets | Recurring questions or complaints about conversion flows |
+| Heatmaps/recordings | Where users hesitate, rage-click, or abandon |
+| Past experiments | "Significant loser" tests often reveal new angles to try |
+
+### ICE Prioritization
+
+Score each hypothesis 1-10 on three dimensions:
+
+| Dimension | Question |
+|-----------|----------|
+| **Impact** | If this works, how much will it move the primary metric? |
+| **Confidence** | How sure are we this will work? (Based on data, not gut.) |
+| **Ease** | How fast and cheap can we ship and measure this? |
+
+**ICE Score** = (Impact + Confidence + Ease) / 3
+
+Run highest-scoring experiments first. Re-score monthly as context changes.
+
+### Experiment Velocity
+
+Track your experimentation rate as a leading indicator of growth:
+
+| Metric | Target |
+|--------|--------|
+| Experiments launched per month | 4-8 for most teams |
+| Win rate | 20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses) |
+| Average test duration | 2-4 weeks |
+| Backlog depth | 20+ hypotheses queued |
+| Cumulative lift | Compound gains from all winners |
+
+### The Experiment Playbook
+
+When a test wins, don't just implement it — document the pattern:
+
+```
+## [Experiment Name]
+**Date**: [date]
+**Hypothesis**: [the hypothesis]
+**Sample size**: [n per variant]
+**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
+**Guardrails**: [any guardrail metrics and their outcomes]
+**Segment deltas**: [notable differences by device, segment, or cohort]
+**Why it worked/failed**: [analysis]
+**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
+**Apply to**: [other pages/flows where this pattern might work]
+**Status**: [implemented / parked / needs follow-up test]
+```
+
+Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.
+
+### Experiment Cadence
+
+**Weekly (30 min)**: Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.
+
+**Bi-weekly**: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.
+
+**Monthly (1 hour)**: Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.
+
+**Quarterly**: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?
+
+---
+
 ## Common Mistakes

 ### Test Design
@@ -2,7 +2,7 @@
 name: social-content
 description: "When the user wants help creating, scheduling, or optimizing social media content for LinkedIn, Twitter/X, Instagram, TikTok, Facebook, or other platforms. Also use when the user mentions 'LinkedIn post,' 'Twitter thread,' 'social media,' 'content calendar,' 'social scheduling,' 'engagement,' 'viral content,' 'what should I post,' 'repurpose this content,' 'tweet ideas,' 'LinkedIn carousel,' 'social media strategy,' or 'grow my following.' Use this for any social media content creation, repurposing, or scheduling task. For broader content strategy, see content-strategy."
 metadata:
-  version: 1.2.0
+  version: 1.1.0
 ---

 # Social Content
@@ -107,7 +107,7 @@ The first line determines whether anyone reads the rest.

 ## Content Repurposing System

-Turn one piece of content into many. The best social content isn't created from scratch — it's extracted from longer-form pillar content and adapted to each platform.
+Turn one piece of content into many:

 ### Blog Post → Social Content

@@ -119,61 +119,13 @@ Turn one piece of content into many. The best social content isn't created from
 | Instagram | Carousel with visuals |
 | Instagram | Reel summarizing the post |

-### Podcast / Video → Social Content
-
-Extract "content atoms" — self-contained moments from any long-form content that work on their own:
-
-| Atom Type | What to Look For | Best Platform |
-|-----------|-----------------|---------------|
-| Quotable moment | A bold claim, hot take, or memorable line (15-60 sec) | Twitter/X, LinkedIn, TikTok |
-| Story arc | A complete mini-story with setup, conflict, resolution (60-90 sec) | Instagram Reels, TikTok, YouTube Shorts |
-| Tactical tip | A specific how-to or framework explained clearly (30-60 sec) | LinkedIn, YouTube Shorts |
-| Controversial take | A contrarian opinion that sparks debate | Twitter/X, LinkedIn |
-| Data/stat callout | A surprising number or research finding | LinkedIn carousel, Twitter/X |
-| Behind-the-scenes | Authentic, unpolished moments | Instagram Stories, TikTok |
-
-**Podcast repurposing workflow:**
-1. **Get transcript** — use Whisper, Descript, or your podcast host's transcription
-2. **Mark timestamps** — flag the 5-10 best moments while listening or scanning transcript
-3. **Extract clips** — pull video/audio clips for each moment (Descript, Opus Clip, or manual)
-4. **Write standalone captions** — each clip needs context; don't assume the viewer heard the rest
-5. **Add subtitles** — most social video is watched without sound
-6. **Schedule across 1-2 weeks** — spread a single episode across multiple posts
-
-**Per episode, aim for:**
- 3-5 short video clips or audiograms (15-60 sec) for Reels/TikTok/Shorts
- 1-2 LinkedIn text posts from key insights
- 1 Twitter/X thread of takeaways
- 1 carousel summarizing the main framework or list
- 1 newsletter section or blog post from the best segment
-
-### Webinar / Live Event → Social Content
-
-| Extract | Format |
-|---------|--------|
-| Key slides with commentary | LinkedIn carousel |
-| Q&A highlights | Twitter/X thread |
-| Speaker quotes | Quote graphics for Instagram/LinkedIn |
-| Audience reactions/poll results | Engagement posts |
-| Full recording → short clips | Reels, TikTok, Shorts |
-
-### Newsletter → Social Content
-
-| Extract | Format |
-|---------|--------|
-| Main insight | LinkedIn post |
-| Curated links with commentary | Twitter/X thread |
-| Data or stat | Quote graphic |
-| Hot take or opinion | Twitter/X post, LinkedIn |
-
 ### Repurposing Workflow

-1. **Create pillar content** (blog, video, podcast, webinar, newsletter)
-2. **Extract content atoms** (5-10 per piece — quotes, stories, tips, data)
-3. **Adapt to each platform** (format, length, and tone)
-4. **Write standalone captions** (each post must work without context)
-5. **Schedule across the week** (spread distribution, don't dump all at once)
-6. **Update and reshare** (evergreen content can repeat every 3-6 months)
+1. **Create pillar content** (blog, video, podcast)
+2. **Extract key insights** (3-5 per piece)
+3. **Adapt to each platform** (format and tone)
+4. **Schedule across the week** (spread distribution)
+5. **Update and reshare** (evergreen content can repeat)

 ---