Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Corey Haines
|
b902ed2d79 |
@@ -1,8 +1,8 @@
|
||||
---
|
||||
name: ab-test-setup
|
||||
description: When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program," or "experiment playbook." Use this whenever someone is comparing two approaches and wants to measure which performs better, or when they want to build a systematic experimentation practice. For tracking implementation, see analytics-tracking. For page-level conversion optimization, see page-cro.
|
||||
description: When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," or "how long should I run this test." Use this whenever someone is comparing two approaches and wants to measure which performs better. For tracking implementation, see analytics-tracking. For page-level conversion optimization, see page-cro.
|
||||
metadata:
|
||||
version: 1.2.0
|
||||
version: 1.1.0
|
||||
---
|
||||
|
||||
# A/B Test Setup
|
||||
@@ -229,93 +229,6 @@ Document every test with:
|
||||
|
||||
---
|
||||
|
||||
## Growth Experimentation Program
|
||||
|
||||
Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.
|
||||
|
||||
### The Experiment Loop
|
||||
|
||||
```
|
||||
1. Generate hypotheses (from data, research, competitors, customer feedback)
|
||||
2. Prioritize with ICE scoring
|
||||
3. Design and run the test
|
||||
4. Analyze results with statistical rigor
|
||||
5. Promote winners to a playbook
|
||||
6. Generate new hypotheses from learnings
|
||||
→ Repeat
|
||||
```
|
||||
|
||||
### Hypothesis Generation
|
||||
|
||||
Feed your experiment backlog from multiple sources:
|
||||
|
||||
| Source | What to Look For |
|
||||
|--------|-----------------|
|
||||
| Analytics | Drop-off points, low-converting pages, underperforming segments |
|
||||
| Customer research | Pain points, confusion, unmet expectations |
|
||||
| Competitor analysis | Features, messaging, or UX patterns they use that you don't |
|
||||
| Support tickets | Recurring questions or complaints about conversion flows |
|
||||
| Heatmaps/recordings | Where users hesitate, rage-click, or abandon |
|
||||
| Past experiments | "Significant loser" tests often reveal new angles to try |
|
||||
|
||||
### ICE Prioritization
|
||||
|
||||
Score each hypothesis 1-10 on three dimensions:
|
||||
|
||||
| Dimension | Question |
|
||||
|-----------|----------|
|
||||
| **Impact** | If this works, how much will it move the primary metric? |
|
||||
| **Confidence** | How sure are we this will work? (Based on data, not gut.) |
|
||||
| **Ease** | How fast and cheap can we ship and measure this? |
|
||||
|
||||
**ICE Score** = (Impact + Confidence + Ease) / 3
|
||||
|
||||
Run highest-scoring experiments first. Re-score monthly as context changes.
|
||||
|
||||
### Experiment Velocity
|
||||
|
||||
Track your experimentation rate as a leading indicator of growth:
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| Experiments launched per month | 4-8 for most teams |
|
||||
| Win rate | 20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses) |
|
||||
| Average test duration | 2-4 weeks |
|
||||
| Backlog depth | 20+ hypotheses queued |
|
||||
| Cumulative lift | Compound gains from all winners |
|
||||
|
||||
### The Experiment Playbook
|
||||
|
||||
When a test wins, don't just implement it — document the pattern:
|
||||
|
||||
```
|
||||
## [Experiment Name]
|
||||
**Date**: [date]
|
||||
**Hypothesis**: [the hypothesis]
|
||||
**Sample size**: [n per variant]
|
||||
**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
|
||||
**Guardrails**: [any guardrail metrics and their outcomes]
|
||||
**Segment deltas**: [notable differences by device, segment, or cohort]
|
||||
**Why it worked/failed**: [analysis]
|
||||
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
|
||||
**Apply to**: [other pages/flows where this pattern might work]
|
||||
**Status**: [implemented / parked / needs follow-up test]
|
||||
```
|
||||
|
||||
Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.
|
||||
|
||||
### Experiment Cadence
|
||||
|
||||
**Weekly (30 min)**: Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.
|
||||
|
||||
**Bi-weekly**: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.
|
||||
|
||||
**Monthly (1 hour)**: Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.
|
||||
|
||||
**Quarterly**: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
### Test Design
|
||||
|
||||
@@ -121,17 +121,18 @@ Choose sources based on your ICP type — then read `references/source-guides.md
|
||||
|
||||
| ICP Type | Primary Sources |
|
||||
|----------|----------------|
|
||||
| B2B SaaS / technical buyers | Reddit (role-specific subs), G2/Capterra, Hacker News, LinkedIn, Indie Hackers |
|
||||
| SMB / founders | Reddit (r/entrepreneur, r/smallbusiness), Indie Hackers, Product Hunt, Facebook Groups |
|
||||
| B2B SaaS / technical buyers | Reddit (role-specific subs), G2/Capterra, Hacker News, LinkedIn, Indie Hackers, SparkToro |
|
||||
| SMB / founders | Reddit (r/entrepreneur, r/smallbusiness), Indie Hackers, Product Hunt, Facebook Groups, SparkToro |
|
||||
| Developer / DevOps | r/devops, r/programming, Hacker News, Stack Overflow, Discord servers |
|
||||
| B2C / consumer | App store reviews (1-3 star), Reddit hobby/lifestyle subs, YouTube comments, TikTok/Instagram comments |
|
||||
| Enterprise | LinkedIn, industry analyst reports, G2 Enterprise filter, job postings |
|
||||
| Enterprise | LinkedIn, industry analyst reports, G2 Enterprise filter, job postings, SparkToro |
|
||||
|
||||
**Quick decision guide:**
|
||||
- Have a product category? → Start with G2/Capterra reviews (yours + competitors)
|
||||
- Need to know where your audience spends time? → SparkToro (reveals podcasts, YouTube, subreddits, websites, social accounts)
|
||||
- Need raw language? → Reddit and YouTube comments
|
||||
- Need trigger events? → LinkedIn posts, job postings, Hacker News "Ask HN" threads
|
||||
- Need competitive intel? → Competitor 4-star reviews on G2; Product Hunt discussions
|
||||
- Need competitive intel? → Competitor 4-star reviews on G2; Product Hunt discussions; SparkToro competitor audience analysis
|
||||
|
||||
### What to Extract from Each Source
|
||||
|
||||
|
||||
@@ -282,6 +282,61 @@ Comments on review videos are especially valuable — these are people actively
|
||||
|
||||
---
|
||||
|
||||
## SparkToro (Audience Intelligence)
|
||||
|
||||
SparkToro is a behavioral audience research tool. Instead of mining individual posts and comments, it aggregates clickstream, search, and social data to show what your audience does at scale — what they read, watch, listen to, follow, and search for.
|
||||
|
||||
### When to Use SparkToro vs. Manual Research
|
||||
|
||||
- **SparkToro first** when you need to understand where your ICP spends time, what content they consume, and which influencers they follow — it answers these questions in seconds with aggregated data
|
||||
- **Manual research first** (Reddit, G2, communities) when you need raw language, exact quotes, emotional context, and the "why" behind behavior
|
||||
- **Best together**: Use SparkToro to identify which podcasts, subreddits, and websites matter, then go mine those sources manually for voice-of-customer language
|
||||
|
||||
### Key Queries to Run
|
||||
|
||||
**By competitor:**
|
||||
- "People who follow @competitor" — reveals shared audience affinities
|
||||
- "People who visit competitor.com" — shows what else they consume
|
||||
|
||||
**By audience description:**
|
||||
- "People who frequently talk about [topic]" — finds audience behaviors
|
||||
- "People whose bio contains [job title]" — profiles a role-based segment
|
||||
|
||||
**By your own audience:**
|
||||
- "People who visit yourdomain.com" — understand your actual audience
|
||||
- Compare against competitor audience profiles to find gaps
|
||||
|
||||
### What to Extract
|
||||
|
||||
| Data Type | What It Tells You | Use It For |
|
||||
|-----------|------------------|------------|
|
||||
| Top websites visited | Where your audience reads | Content partnerships, guest posting targets |
|
||||
| Top podcasts | What they listen to | Podcast guesting, sponsorship decisions |
|
||||
| Top YouTube channels | What they watch | Video content strategy, ad placements |
|
||||
| Top subreddits | Where they discuss | Community participation, Reddit ad targeting |
|
||||
| Search keywords | What they Google | SEO and content topic planning |
|
||||
| AI prompt topics | What they ask AI tools | Emerging content opportunities |
|
||||
| Social accounts followed | Who influences them | Influencer partnerships, co-marketing |
|
||||
| Demographics | Who they are | Persona building, ad targeting |
|
||||
|
||||
### Source Weighting
|
||||
|
||||
SparkToro data is aggregated and anonymized — it shows patterns, not individual opinions. Treat it as:
|
||||
- **High confidence** for behavioral data (what they visit, follow, search for)
|
||||
- **Medium confidence** for demographic data (self-reported, may be incomplete)
|
||||
- **Not a substitute** for qualitative research (doesn't capture language, emotions, or the "why")
|
||||
|
||||
### Limitations
|
||||
|
||||
- Free tier: 5 reports/month, shallow results (top 5–10)
|
||||
- No public API — all research done through web interface
|
||||
- Skews English-language, US-centric
|
||||
- Shows what audiences do, not why — pair with qualitative sources
|
||||
|
||||
See [tools/integrations/sparktoro.md](../../../tools/integrations/sparktoro.md) for full tool details and pricing.
|
||||
|
||||
---
|
||||
|
||||
## Organizing Your Research
|
||||
|
||||
Use a simple tagging system across all sources:
|
||||
@@ -319,6 +374,7 @@ Not all sources carry equal weight. Use this guide when assigning confidence lab
|
||||
| Survey (multiple choice) | Low-medium | Artifacts of the options you provided |
|
||||
| NPS verbatims | Medium | Correlates with score; prompted by the survey moment |
|
||||
| YouTube/TikTok comments | Medium | Skews toward engaged viewers; social performance |
|
||||
| SparkToro audience data | Medium-high | Aggregated behavioral data; strong for "what" but not "why" |
|
||||
| Job postings | Low-medium | Aspirational, not necessarily reflective of current pain |
|
||||
|
||||
### Confidence Labels in Practice
|
||||
|
||||
@@ -74,6 +74,7 @@ Quick reference for AI agents to discover tool capabilities and integration meth
|
||||
| pendo | Product Analytics | ✓ | - | [✓](clis/pendo.js) | - | [pendo.md](integrations/pendo.md) |
|
||||
| similarweb | Competitive Intelligence | ✓ | - | [✓](clis/similarweb.js) | - | [similarweb.md](integrations/similarweb.md) |
|
||||
| firehose | Competitive Intelligence | ✓ | - | - | - | [firehose.md](integrations/firehose.md) |
|
||||
| sparktoro | Audience Research | - | - | - | - | [sparktoro.md](integrations/sparktoro.md) |
|
||||
| airops | AI Content | ✓ | - | [✓](clis/airops.js) | - | [airops.md](integrations/airops.md) |
|
||||
| buffer | Social | ✓ | - | [✓](clis/buffer.js) | - | [buffer.md](integrations/buffer.md) |
|
||||
| wistia | Video | ✓ | - | [✓](clis/wistia.js) | - | [wistia.md](integrations/wistia.md) |
|
||||
@@ -340,6 +341,16 @@ Traffic analytics, competitor benchmarking, and market research.
|
||||
|
||||
**Agent recommendation**: Similarweb for competitor traffic analysis and market benchmarking.
|
||||
|
||||
### Audience Research
|
||||
|
||||
Audience intelligence and behavioral research tools.
|
||||
|
||||
| Tool | Best For | Notes |
|
||||
|------|----------|-------|
|
||||
| **sparktoro** | Audience affinities, behavioral data | Clickstream + social data |
|
||||
|
||||
**Agent recommendation**: SparkToro for discovering where your ICP spends time — what they read, watch, listen to, follow, and search for. Essential for customer research, content strategy, and media buying decisions.
|
||||
|
||||
### AI Content
|
||||
|
||||
AI-powered content generation and optimization platforms.
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# SparkToro
|
||||
|
||||
Audience research platform that reveals what your target audience reads, watches, listens to, follows, and searches for — using clickstream data, Google search data, and public social profiles.
|
||||
|
||||
## Capabilities
|
||||
|
||||
| Integration | Available | Notes |
|
||||
|-------------|-----------|-------|
|
||||
| API | - | Not yet public (coming soon) |
|
||||
| MCP | - | Not available |
|
||||
| CLI | - | Not available |
|
||||
| SDK | - | Not available |
|
||||
|
||||
SparkToro is primarily a web-based research tool. No public API, CLI, or SDK is currently available. Use the web interface at https://sparktoro.com for all queries.
|
||||
|
||||
## Authentication
|
||||
|
||||
- **Type**: Account login at https://sparktoro.com
|
||||
- **Free tier**: 5 reports/month with limited results
|
||||
- **Paid plans**: $50–$300/month with expanded results and exports
|
||||
|
||||
## Pricing Tiers
|
||||
|
||||
| Plan | Price | Reports/Month | Users | Result Depth |
|
||||
|------|-------|---------------|-------|-------------|
|
||||
| Free | $0 | 5 | 1 | Top 5–10 results |
|
||||
| Personal | $50/mo | 50 | 1 | Top 50 results |
|
||||
| Business | $150/mo | 500 | 10 | Top 150 results, contact data, AI advice |
|
||||
| Agency | $300/mo | Unlimited | 100 | Top 300 results, full CSV export |
|
||||
|
||||
## What SparkToro Reveals
|
||||
|
||||
### Audience Behaviors
|
||||
- **Websites** they visit and engage with
|
||||
- **Podcasts** they listen to
|
||||
- **YouTube channels** they watch
|
||||
- **Subreddits** they participate in
|
||||
- **Social accounts** they follow
|
||||
- **Search keywords** they use on Google
|
||||
- **AI prompt topics** they ask ChatGPT, Claude, Gemini
|
||||
|
||||
### Audience Demographics
|
||||
- Gender, age ranges
|
||||
- Job titles and roles
|
||||
- Industries and skills
|
||||
- Education levels
|
||||
- Geographic distribution
|
||||
- Interests and affinities
|
||||
|
||||
### Audience Characteristics
|
||||
- Bio descriptions and self-identifiers
|
||||
- Language patterns in posts and comments
|
||||
- Preferred social networks and platforms
|
||||
- E-commerce platforms they use
|
||||
|
||||
## Common Agent Operations
|
||||
|
||||
Since SparkToro has no API, these are the research workflows agents should guide users through.
|
||||
|
||||
### Audience Profile Research
|
||||
|
||||
Query SparkToro with phrases like:
|
||||
- "People who follow @competitor" — reveals shared audience behaviors
|
||||
- "People who visit competitor.com" — shows what else they consume
|
||||
- "People who frequently talk about [topic]" — finds audience affinities
|
||||
- "People whose bio contains [job title]" — profiles a role-based segment
|
||||
|
||||
### Finding Where Your ICP Spends Time
|
||||
|
||||
1. Search for your ICP by description, competitor followers, or website visitors
|
||||
2. Extract: top websites visited, podcasts listened to, YouTube channels watched, subreddits
|
||||
3. Use this to prioritize: guest podcast appearances, content partnerships, ad placements, community participation
|
||||
|
||||
### Discovering Content Topics
|
||||
|
||||
1. Search your audience segment
|
||||
2. Review the "Search Keywords" tab — what they Google
|
||||
3. Review the "AI Prompt Topics" tab — what they ask AI tools
|
||||
4. Use these to inform content strategy and SEO keyword targeting
|
||||
|
||||
### Building Data-Backed Personas
|
||||
|
||||
1. Run 3–5 queries for different segments of your audience
|
||||
2. Compare demographic breakdowns across segments
|
||||
3. Note which behaviors and affinities are shared vs. unique per segment
|
||||
4. Export data and build personas grounded in observed behavior, not assumptions
|
||||
|
||||
### Competitive Audience Analysis
|
||||
|
||||
1. Search "People who follow @competitor" or "People who visit competitor.com"
|
||||
2. Compare against your own audience profile
|
||||
3. Identify: channels they use that you don't, content they consume that you don't produce, influencers they follow that you haven't engaged
|
||||
|
||||
## Data Sources
|
||||
|
||||
SparkToro aggregates from three sources:
|
||||
- **Clickstream data** — anonymized browsing behavior
|
||||
- **Google search results** — search keyword patterns
|
||||
- **Public social profiles** — bios, follows, engagement
|
||||
|
||||
## When to Use
|
||||
|
||||
- Identifying where your ICP spends time online (podcasts, YouTube, subreddits, websites)
|
||||
- Finding influencers and social accounts your audience follows
|
||||
- Discovering content topics and search keywords your audience cares about
|
||||
- Building data-backed personas instead of assumption-based ones
|
||||
- Planning podcast guest appearances, sponsorships, or content partnerships
|
||||
- Understanding what your competitors' audience looks like
|
||||
- Validating audience assumptions with behavioral data
|
||||
- Discovering AI prompt topics your audience uses
|
||||
|
||||
## Limitations
|
||||
|
||||
- No public API — all research is done through the web interface
|
||||
- Free tier limited to 5 reports/month with shallow results
|
||||
- Data skews toward English-language, US-centric audiences
|
||||
- Clickstream data may not capture all niche audiences
|
||||
- Cannot track individual users — all data is aggregated and anonymized
|
||||
|
||||
## Relevant Skills
|
||||
|
||||
- customer-research
|
||||
- content-strategy
|
||||
- competitor-alternatives
|
||||
- paid-ads
|
||||
- social-content
|
||||
- cold-email
|
||||
Reference in New Issue
Block a user