From 2b4cccbfcf5c9f8e986b95b97d7f216fadbfba37 Mon Sep 17 00:00:00 2001
From: njcameron <1101879+njcameron@users.noreply.github.com>
Date: Tue, 7 Apr 2026 12:07:48 +0000
Subject: [PATCH] feat: persist raw scrape, SEO, and review data to disk

---
 PR_DESCRIPTION.md                    | 30 ++++++++++++++++++++++
 skills/competitor-profiling/SKILL.md | 37 ++++++++++++++++++++++++++--
 2 files changed, 65 insertions(+), 2 deletions(-)
 create mode 100644 PR_DESCRIPTION.md

diff --git a/PR_DESCRIPTION.md b/PR_DESCRIPTION.md
new file mode 100644
index 0000000..f4a7b56
--- /dev/null
+++ b/PR_DESCRIPTION.md
@@ -0,0 +1,30 @@
+# Add competitor-profiling skill
+
+Adds a new `competitor-profiling` skill for researching and profiling competitors from a list of URLs. The skill combines live site scraping (Firecrawl) with SEO and market intelligence (DataForSEO) to produce structured, comparable competitor profile documents.
+
+## What it does
+
+Given competitor URLs, the skill:
+
+1. Maps each competitor's site and scrapes key pages (homepage, pricing, features, about, customers, integrations, changelog)
+2. Pulls SEO and market data — domain authority, backlinks, ranked keywords, organic traffic, top pages, and adjacent competitors
+3. Optionally mines review sites (G2, Capterra, Product Hunt, TrustRadius) for sentiment and themes
+4. Synthesizes everything into a consistent profile template, plus a cross-competitor summary doc
+
+Supports both **quick scan** (homepage + pricing + basic SEO) and **deep profile** (full scrape + backlink analysis + review mining) modes.
+
+## Files
+
+- `skills/competitor-profiling/SKILL.md` — main skill instructions
+- `skills/competitor-profiling/references/templates.md` — full profile + summary templates
+- `skills/competitor-profiling/references/tool-reference.md` — Firecrawl + DataForSEO MCP tool reference with example calls
+
+## Why it's useful
+
+Fills the gap between raw competitive research and downstream skills. The structured output is designed to feed directly into:
+
+- `competitor-alternatives` — comparison/alternative pages
+- `sales-enablement` — battle cards
+- `content-strategy` — content gap analysis
+- `pricing-strategy` — pricing teardown
+- `seo-audit` — relative SEO benchmarking
diff --git a/skills/competitor-profiling/SKILL.md b/skills/competitor-profiling/SKILL.md
index 3ba3fd1..bded2d5 100644
--- a/skills/competitor-profiling/SKILL.md
+++ b/skills/competitor-profiling/SKILL.md
@@ -41,6 +41,37 @@ Don't exaggerate competitor weaknesses or downplay their strengths. Accurate pro
 
 ---
 
+## Saving Raw Data
+
+Before synthesizing the profile, persist all raw scrape, SEO, and review data to disk so it can be re-read, audited, or re-used later without re-running expensive API calls.
+
+**Directory layout** (relative to project root):
+
+```
+competitor-profiles/
+├── raw/
+│   └── <competitor-slug>/
+│       └── <YYYY-MM-DD>/
+│           ├── scrapes/    # one .md file per scraped page (homepage.md, pricing.md, ...)
+│           ├── seo/        # one .json file per DataForSEO call (backlinks-summary.json, ranked-keywords.json, ...)
+│           └── reviews/    # one .md or .json file per review source (g2.md, capterra.md, ...)
+├── <competitor-slug>.md    # final synthesized profile
+└── _summary.md             # cross-competitor summary
+```
+
+Rules:
+
+- `<competitor-slug>` is lowercase, hyphenated (e.g. `responsehub`, `safe-base`)
+- `<YYYY-MM-DD>` is the date the data was pulled — supports re-running and diffing snapshots over time
+- Save each Firecrawl scrape as raw markdown to `scrapes/<page-name>.md`
+- Save each DataForSEO response as raw JSON to `seo/<endpoint-name>.json`
+- Save each review source to `reviews/<source>.md` (cleaned text) or `.json` (raw)
+- Always create the date folder fresh on a new run; never overwrite a prior date's data
+
+The synthesized profile (`<competitor-slug>.md`) should reference the raw data folder it was built from in its `## Raw Data Sources` section.
+
+---
+
 ## Research Process
 
 ### Phase 1: Site Scraping (Firecrawl)
@@ -73,6 +104,8 @@ Use **Firecrawl Scrape** on each identified page:
 firecrawl_scrape → each key page URL
 ```
 
+Save each result to `competitor-profiles/raw/<competitor-slug>/<YYYY-MM-DD>/scrapes/<page-name>.md` before extracting fields.
+
 Extract from each page:
 
 | Page | What to Extract |
@@ -93,13 +126,13 @@ Use **Firecrawl Scrape** or **Firecrawl Search** to find:
 - Product Hunt launch page
 - TrustRadius profile
 
-Extract: overall rating, review count, common praise themes, common complaint themes, and 3-5 representative quotes.
+Save each scraped review page to `competitor-profiles/raw/<competitor-slug>/<YYYY-MM-DD>/reviews/<source>.md`. Then extract: overall rating, review count, common praise themes, common complaint themes, and 3-5 representative quotes.
 
 ---
 
 ### Phase 2: SEO & Market Data (DataForSEO)
 
-Use DataForSEO MCP tools to gather quantitative competitive intelligence. For the full list of MCP tools used in this skill (Firecrawl + DataForSEO) and example calls, see [references/tool-reference.md](references/tool-reference.md).
+Use DataForSEO MCP tools to gather quantitative competitive intelligence. Save each raw response as JSON to `competitor-profiles/raw/<competitor-slug>/<YYYY-MM-DD>/seo/<endpoint-name>.json` before parsing it into the profile. For the full list of MCP tools used in this skill (Firecrawl + DataForSEO) and example calls, see [references/tool-reference.md](references/tool-reference.md).
 
 #### Domain Authority & Backlinks