diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/00-setup.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/00-setup.md new file mode 100644 index 000000000..14fd08cfd --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/00-setup.md @@ -0,0 +1,108 @@ +# Phase 0 + 1 — Environment Assessment & Journal Setup + +Before a debugger touches anything, you need a map of what's running and a ledger of what you'll touch. Skipping either phase is how debug sessions turn into "why is my repo dirty a week later" sessions. + +--- + +## Phase 0 — Environment Assessment + +Map the ground truth before you attach. Attaching the wrong way wastes the first hour. + +### 1. Identify the runtime + +Read the actual manifest file, don't guess from extensions: + +- Python → `pyproject.toml`, `requirements*.txt`, `setup.py`, `uv.lock`, `.python-version` +- Node → `package.json` (check `scripts`, check `engines`, check `type: module`) +- Rust → `Cargo.toml`, `rust-toolchain*` +- Go → `go.mod`, `go.sum` +- Native / mixed → `Makefile`, `CMakeLists.txt`, the binary itself (`file `) + +### 2. Load the matching runtime reference + +The moment you know the runtime, open `references/runtimes/.md`. The commands in this phase (and every phase after) are runtime-specific. The shape of the answers is the same; the commands are not. + +### 3. Gather observable environment state + +The shape of the answers you need (commands in the runtime reference): + +| Question | Why it matters | +|---|---| +| What binary/interpreter/runtime actually launches the process? | Determines debugger flag plumbing. Wrappers (`tsx`, `poetry run`, `cargo run`, `bun`, supervisor scripts) change how flags propagate. | +| Is there already a debug-relevant port in use, or another instance of the service running? | Either attach to it or kill it deliberately — never silently compete. | +| Are symbols / source maps / debug info present and correct? | This determines whether breakpoints land on the right lines. Compiled-but-not-debug builds, stripped binaries, and incomplete source maps all silently misplace breakpoints. | +| Does the code path require env vars, config files, or auth tokens to reach the bug? | Missing env often produces early-return paths that masquerade as the bug itself. | +| Is there an existing failing test or known repro? | Prefer amplifying an existing repro over inventing one. | +| Are watchers (file watchers, hot reloaders, supervisors) going to restart the process mid-session? | If yes, turn them off before attaching. Restarts drop inspector connections and invalidate breakpoints. | + +### 4. Gate check + +If any answer is "I'm not sure", you are not ready for Phase 1. Investigate until certain. Guessing here cascades into false-positive hypotheses in Phase 2. + +--- + +## Phase 1 — Journal Setup + +Open **one** journal file at the project root: `.debug-journal.md`. Single source of truth for every artifact this skill creates. The contract with the user that you can undo everything. + +### Exclude from git (don't pollute the committed ignore list) + +```bash +grep -qx '.debug-journal.md' .git/info/exclude || echo '.debug-journal.md' >> .git/info/exclude +``` + +`.git/info/exclude` is per-clone and not committed — perfect for local-session artifacts. + +### Journal template + +```markdown +# Debug Journal — +Started: +Goal: + +## Environment snapshot (Phase 0) +- Runtime: +- Entry: +- Ports / sockets: +- Git HEAD: , working tree clean? +- References read: + +## Hypotheses +1. [STATUS] — distinguishing evidence: — if true, fix is: +2. ... + +## Failed hypothesis round counter +- Round 1: +- Round 2: + + +## Artifacts to revert + +- [ ] `src/foo.py` — added `breakpoint()` on 2 lines. Revert: `git checkout src/foo.py` +- [ ] tmux session `debug-server`. Kill: `tmux kill-session -t debug-server` +- [ ] `/tmp/debug-payload.json`. Remove: `rm /tmp/debug-payload.json` +- [ ] env var in current shell: `FOO_BASE_URL=...`. Unset when done. +- [ ] GDB session save: `~/ghidra-projects/scratch.gzf`. Remove if not promoting. + +## Findings + + +## Oracle Triple (if invoked) + + +## Final fix + +``` + +### The journal-then-modify rule + +Before any modification to the repo, shell, or system state, append to "Artifacts to revert" first. This one discipline is what prevents debug sessions from becoming git cleanup sessions. + +If you catch yourself about to run a command that creates a file, opens a port, or modifies source — stop, journal the intended artifact with its revert command, then run the command. Not the other way around. + +### Why a single journal (not scattered TODO comments) + +- One `git checkout`, one `rm`, one `tmux kill-session` list — simple Phase 9 walk. +- Survives interruptions. If you get pulled away mid-session, the next agent (or you later) can continue or revert without guessing. +- Prevents the most common failure: leaving `console.log`/`print()`/`dbg!` scattered across the tree. diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/02-investigate.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/02-investigate.md new file mode 100644 index 000000000..5a01c301e --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/02-investigate.md @@ -0,0 +1,130 @@ +# Phase 2 + 3 — Hypothesis Formation & Parallel Investigation + +One hypothesis is a hunch. Three hypotheses is a decision. Investigation is how you turn the decision into runtime evidence. + +--- + +## Phase 2 — Hypothesis Formation (Minimum Three) + +### Why three, not one + +A single hypothesis creates confirmation bias: you'll read runtime state looking for evidence that confirms it and unconsciously discount contradictions. Three hypotheses force you to design queries that *distinguish* between them, which is the only way runtime evidence becomes decisive. + +### Generate across orthogonal axes + +If your three hypotheses are all variations of "the handler has a bug", you don't actually have three hypotheses. Span the space: + +| Axis | Example framing | +|---|---| +| **User-code logic** | "The handler early-returns because condition X is unexpectedly true" | +| **Library/SDK behavior** | "The third-party client swallows the error and returns a stub" | +| **Environment/config** | "The env var is read at module-load time before it gets populated, so it's empty" | +| **Async/timing** | "The promise rejects (or goroutine panics) after the response is already sent" | +| **Silent side-effect** | "An earlier turn mutated shared state that the current turn inherits" | +| **Observability gap** | "The error is raised but suppressed before logging; it only exists as an unawaited rejection / ignored signal" | +| **Binary-level** (when applicable) | "The function we think is running is actually jumped over by a patched thunk / a different version loaded" | +| **Build-vs-runtime** | "The code we're reading is not the code that's running — stale build, wrong symlink, cached wheel, or dist/ ahead of src/" | + +### For each hypothesis, write in the journal + +1. **Claim** — one sentence. +2. **Distinguishing evidence** — the exact value or state that confirms or refutes it, AND where to read it (file:line, log source, breakpoint location, memory address). +3. **If true, the fix is** — two words. Forces you to think through fix cost before committing to the hunt. + +### Collapse rule + +If two hypotheses have identical distinguishing evidence, they aren't actually different — collapse them and find a real alternative. If you can't come up with a third distinct hypothesis, you don't understand the system well enough yet. Go read a little more code before investigating. + +--- + +## Phase 3 — Parallel Investigation + +Branch depending on what's available. + +### Path A: Team mode ENABLED + +When the `team_*` tools are present, create a **debug-squad** team and split investigation across members working on different evidence sources. This is the right default whenever you have ≥3 hypotheses and any of them would take >10 minutes to investigate single-threaded. + +**Team spec** — write to `~/.omo/teams/debug-squad/config.json`: + +```json +{ + "name": "debug-squad", + "lead": { "kind": "subagent_type", "subagent_type": "sisyphus" }, + "members": [ + { + "kind": "category", + "category": "deep", + "prompt": "You are the Runtime State Inspector. Your job: attach to the live process, hit breakpoints, read program state (variables, heap, goroutines, stack, registers depending on runtime), and report observed values verbatim. Never guess — if you don't see the value, say so. Report back via team_send_message with file:line / address references and captured values. Never edit source code. Never run git commands. If you need an instrumentation statement added (breakpoint(), debugger;, dbg!, etc.), ask the Lead first." + }, + { + "kind": "category", + "category": "deep", + "prompt": "You are the Log Archaeologist. Your job: grep server logs, stderr streams, SDK-internal debug output (DEBUG env, RUST_LOG, GODEBUG, PYTHONASYNCIODEBUG), and correlate timestamps. Produce a timeline of events with latencies. Flag anything that looks like a silent catch, a swallowed rejection, a panic recovered-and-ignored, a success response that contains failure signals (HTTP 200 with empty body, stopReason=error, exit 0 with error-in-stdout). Never edit source code." + }, + { + "kind": "category", + "category": "deep", + "prompt": "You are the Reproduction Engineer. Your job: build the smallest reliable repro — a curl command, a vitest/pytest/go test, a tmux script, a Playwright script for browser bugs, a pwntools script for binary targets. It must reproduce on first try and be copy-pasteable by the Lead. Document exact input, expected output, observed output. Save repro artifacts under /tmp/ and tell the Lead to journal them. If the bug is browser-based you MUST use Playwright CLI — do not simulate with curl." + }, + { + "kind": "category", + "category": "deep", + "prompt": "You are the Trace Correlator. Your job: take findings from the other members and cross-link them. Build a causal chain from symptom to suspected cause. Identify missing evidence. Propose the next single most-decisive runtime query. Never edit source code; only reason across already-captured evidence. If hypotheses diverge sharply after correlation, tell the Lead immediately — that is the signal for the Oracle Triple." + } + ] +} +``` + +**Assignment rule**: one hypothesis → one `team_task_create`. Give each hypothesis to the member whose evidence source is most likely to confirm or refute it. Broadcast the full hypothesis list once via `team_send_message(to="*")` so members know what the others are testing. + +**Lead responsibilities**: +- Maintain the journal (members do not write to it). +- Approve any source-code edits (including `debugger;` / `breakpoint()` / `dbg!` statements). +- Synthesize member reports into updated hypothesis statuses. +- Decide when to disband: `team_shutdown_request` → `team_approve_shutdown` → `team_delete`. + +**Team does NOT include Oracle** — Oracle is a hard-reject team member type. Oracle is used separately in Phase 4 (see `04-oracle-triple.md`). + +### Path B: Team mode DISABLED + +Fan out async explore/deep subagents instead. Same rule: one hypothesis per subagent. + +``` +task(subagent_type="explore", load_skills=[], run_in_background=true, + prompt="[CONTEXT: bug summary + which hypothesis you own + what state to look at] + Runtime state investigation for hypothesis 1: ...") +task(subagent_type="explore", load_skills=[], run_in_background=true, + prompt="Log/timing investigation for hypothesis 2: ...") +task(category="deep", load_skills=[], run_in_background=true, + prompt="Reproduction minimizer for hypothesis 3: ...") +``` + +End your response, wait for completion notifications, then synthesize. + +--- + +## Evidence capture discipline (both paths) + +For every piece of runtime state captured, record in the journal: + +```markdown +### — +- Source: +- Value: `` +- Interpretation: +- Refutes/Confirms: H +``` + +**Verbatim values only. No paraphrasing.** + +- `messages.length=0` is evidence. +- "messages seemed empty" is not evidence — it's a memory of an observation, and memory of observations is where debug sessions go to die. + +If you find yourself about to paraphrase, stop, go back, and copy the raw value. + +--- + +## Round completion + +A "round" is complete when every hypothesis has either confirming or refuting evidence — or when you have exhausted the evidence sources available without a decisive result. If the round ends inconclusively, that counts as a failed round for the counter in the journal. See `04-oracle-triple.md` for what to do at 2 consecutive failed rounds. diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/04-oracle-triple.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/04-oracle-triple.md new file mode 100644 index 000000000..ec094813f --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/04-oracle-triple.md @@ -0,0 +1,136 @@ +# Phase 4 — Oracle Triple Consultation + +At 2 consecutive failed hypothesis rounds, stop investigating and reframe. Continuing past two failures usually means the real cause is in a category you haven't imagined — and more time on your current mental model is wasted time. + +The Oracle Triple is how you break out of the mental box. + +> ⚠️ **Wrong tool for non-debugging tasks.** The Triple is for *stuck root-cause hunts*. If your task is producing an artifact (extraction, reverse engineering, audit, compliance documentation) and you want a skeptical review before declaring it done, use the **Verification Oracle** pattern in [partial-runtime-evidence.md](partial-runtime-evidence.md#verification-oracle-pattern-for-non-debug-tasks). Running the Triple on a finished extraction returns three diverging "what if you tried…" tangents that are not what you need. + +--- + +## When to invoke + +| Situation | Invoke? | +|---|---| +| 1 round failed, you have new distinguishing evidence | No — run one more round with a refined hypothesis set | +| 2 rounds failed, hypotheses now feel like variations of each other | **Yes — invoke now** | +| 2 rounds failed, no new evidence angles left to try | **Yes — invoke now** | +| You've been investigating >2 hours on the same bug | **Yes — invoke now regardless of round count** | +| 1 round failed but the user is watching and wants speed | No — one round isn't enough to justify Oracle cost. Resist the urge. | + +--- + +## Why three Oracles, and why *orthogonal* framings + +A single Oracle call returns a single coherent analysis. Coherent analyses tend to inherit the framing of the prompt, which means they inherit the same blind spots the investigator already has. Three Oracles with *orthogonal framings* force the analyses to diverge, and the places where they agree across frames is where the real signal lives. + +The three framings below are chosen to cover distinct bug-cause categories: + +- **A (obvious-but-missed)** — embarrassingly simple causes the investigator walked past. +- **B (system-boundary)** — causes living at integration seams, not in the code being read. +- **C (invariant-violation)** — assumptions load-bearing to current hypotheses that may themselves be false. + +Spawn all three in parallel. + +--- + +## The three prompts + +``` +task(subagent_type="oracle", load_skills=[], run_in_background=true, + prompt="[CONTEXT: bug description + evidence captured so far, verbatim, with file:line refs] + + Framing A — OBVIOUS-BUT-MISSED. + What is the most embarrassing, most obvious cause that a senior engineer would spot in 30 seconds and we've overlooked? Consider: + - typos, off-by-one + - wrong variable name / wrong constant / wrong import + - stale cache, wrong file edited, wrong process inspected + - attached to the wrong instance of the service + - test harness running different code than the app + - editing src/ while running dist/ + + Give me exactly three candidate causes ranked by likelihood, with one sentence each explaining why our evidence is consistent with each.") + +task(subagent_type="oracle", load_skills=[], run_in_background=true, + prompt="[CONTEXT: bug description + evidence captured so far] + + Framing B — SYSTEM-BOUNDARY. + What if the bug is NOT in the code we've been reading, but at a boundary? Consider: + - third-party SDK behavior that contradicts its docs + - middleware that mutates the request or response + - a proxy/gateway/load balancer that rewrites headers or bodies + - build-time vs runtime env-var resolution + - module-load-order issue + - shared-library version mismatch (system lib vs bundled lib) + - ABI difference (native addons, glibc versions, musl vs glibc) + - wrong transport (HTTP/1.1 vs HTTP/2, TLS version negotiation) + + Give me three candidate causes, each naming the specific boundary and the specific contract assumption that might be violated.") + +task(subagent_type="oracle", load_skills=[], run_in_background=true, + prompt="[CONTEXT: bug description + evidence captured so far] + + Framing C — INVARIANT-VIOLATION. + Which invariants that we've been ASSUMING TRUE might actually be false? + Enumerate the five assumptions most load-bearing to our current hypotheses, then for each: + - describe the smallest runtime query that would falsify it + - predict what the observable would be if the invariant holds vs if it fails + + We want at least one of these queries to be decisive.") +``` + +--- + +## Synthesizing across three Oracles + +**Do not pick the highest-ranked candidate from a single Oracle.** That defeats the purpose of getting three framings. + +Instead, walk the outputs in this order: + +### 1. Agreement scan + +Note which candidate causes appear in at least two Oracles' outputs. Independent agreement across orthogonal framings is strong signal — when the obvious-but-missed framing and the system-boundary framing both land on the same cause, that's usually the bug. + +### 2. Disagreement scan + +Note where Oracles disagree. Disagreement is genuine uncertainty that runtime evidence (not more reasoning) must resolve. Each disagreement becomes a candidate for the next round's distinguishing query. + +### 3. New falsification queries + +Framing C produces concrete "one query that would decide it" suggestions. Pull these verbatim into your new round's evidence-gathering plan — they are designed to be decisive. + +### 4. Build the new hypothesis set + +Minimum 3, same rules as Phase 2. Aim to have hypotheses drawn from the agreement scan (likely cause) AND from the disagreement scan (so one round's evidence resolves the disagreement). + +Record in the journal: + +```markdown +## Oracle Triple — Round +- Invoked at: +- Framing A summary: +- Framing B summary: +- Framing C summary: <5 load-bearing assumptions + falsification queries> + +### Cross-framing agreement +- appeared in A + B +- appeared in B + C + +### New hypothesis set +1. — evidence to gather: +2. ... +``` + +### 5. Reset the counter + +Reset the "consecutive failed rounds" counter to 0. Return to Phase 3 (parallel investigation) with the new set. + +--- + +## If *another* 2 rounds fail after the Oracle Triple + +You are genuinely stuck. This is the escalation threshold. + +Escalate to the user (see `05-escalate.md`) with the full trace: every hypothesis tried, every piece of evidence captured, both Oracle syntheses. Do not guess a fix. + +This is rare — in practice, the Oracle Triple resolves almost all stuck debugging sessions within one round, because it pulls in framings the investigator was too close to the code to see. diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/05-escalate.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/05-escalate.md new file mode 100644 index 000000000..187e9a957 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/05-escalate.md @@ -0,0 +1,69 @@ +# Phase 5 — User Decision Escalation + +Escalation is for genuine ambiguity, not for skipping investigation. Most "should I ask the user" moments are really "I don't want to do one more query" moments, and those are wrong. + +--- + +## Ask the user ONLY when + +- **Evidence exhausted**, contradictions remain, and further investigation would require a decision with policy implications (e.g. "patch the third-party SDK vs wrap it vs change architecture"). +- The bug has **multiple valid fixes with different scope/risk tradeoffs** and the user's preference drives the choice. +- A proposed fix would **change observable product behavior** for the end user (not just fix the internal bug). +- You've **exhausted the Oracle Triple** and another 2 rounds failed after synthesis. + +## Do NOT ask when + +- You haven't tried the Oracle Triple yet. +- The question can be answered by one more runtime query. +- You're asking for permission to do the obvious thing. +- You're asking because you're tired. + +--- + +## Escalation format (paste into the reply) + +Keep it short. Evidence-dense. One decision, not a status update. + +```markdown +## Decision needed + +**What we know** (verbatim evidence, not paraphrase): +- +- +- +- + +**What the decision is** (one sentence): + + +**Options**: + +| # | Fix | Scope | Risk | Effort | +|---|-----|-------|------|--------| +| A | | | | | +| B | ... | ... | ... | ... | +| C | ... | ... | ... | ... | + +**Recommendation**: because . + +Which direction do you want? +``` + +--- + +## Anti-patterns in escalation + +- **Asking without evidence.** "What do you want me to do?" is not an escalation, it's abandonment. Every escalation includes the evidence the user needs to decide. +- **Two questions in one.** One decision per escalation. Multi-part questions lead to partial answers and re-escalation. +- **Escalating before Phase 4.** If you haven't tried the Oracle Triple, you haven't earned the right to escalate. +- **Presenting options you don't actually have.** If option C requires a library the user doesn't use, don't list it. The options are only things you can actually do today. +- **Hiding a recommendation.** The user hired you to think — always end with a recommendation, even if you're low-confidence. Say so explicitly: "Recommendation (low confidence): B, because X. If you have context about Y that I don't, it might change to A." + +--- + +## What happens after the user responds + +- **User picks an option**: return to Phase 6 (root cause confirmation) with the chosen direction. The user's choice is not itself confirmation — you still need runtime evidence that the cause you're fixing is the cause in play. +- **User proposes a different option you hadn't considered**: treat it as new information. Update hypotheses. May trigger another Phase 3 round. +- **User gives more context that resolves the disagreement**: skip to Phase 6. +- **User is also unsure**: that's a signal you need more evidence, not more opinions. Run one more targeted query before asking again. diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/06-fix.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/06-fix.md new file mode 100644 index 000000000..70e36da85 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/06-fix.md @@ -0,0 +1,116 @@ +# Phase 6 + 7 — Root Cause Confirmation & TDD Fix + +A cause is not "confirmed" until you can toggle the bug by toggling the cause. Every other level of evidence is correlation, and correlation-driven fixes ship bugs. + +--- + +## Phase 6 — Root Cause Confirmation + +You are allowed to call the cause "confirmed" only when ALL THREE of these hold: + +### 1. Captured runtime value matches the hypothesis exactly + +Not "the value looks consistent with" — the value is exactly the value the hypothesis predicted. If your hypothesis was "baseUrl is api.anthropic.com despite ANTHROPIC_BASE_URL being set to a proxy", the captured value is literally `"https://api.anthropic.com"` in the debugger at the moment of the HTTP call. + +### 2. Reproducible + +Running the repro a second time yields the same observation. Flaky repros mean you haven't isolated the cause; you've isolated a symptom that sometimes appears when the cause does. Keep investigating. + +### 3. Toggle proof (the one most skipped) + +**Changing the value** (via debugger assignment, env override, or a speculative one-line patch) **makes the bug disappear — and reverting brings the bug back**. + +If you can't toggle the bug by toggling the suspected cause, what you have is a correlation, not a mechanism. A correlation is a strong hypothesis, not a confirmed cause. + +Examples of a valid toggle proof: + +| Suspected cause | Toggle | +|---|---| +| Env var overrides library default, and the override is wrong | Unset the env var → bug goes away. Reset it → bug comes back. | +| Async task is not awaited | Add `await` → bug goes away. Remove `await` → bug comes back. | +| Third-party SDK uses hardcoded URL | Monkey-patch SDK to use env URL → bug goes away. Unpatch → bug comes back. | +| Race condition on shared state | Add a mutex → bug goes away under load. Remove mutex → bug comes back under load. | + +If you can't construct a toggle proof, you haven't confirmed the cause. Run one more round. + +### Update the journal + +```markdown +## Root cause (confirmed ) +- Mechanism: +- Evidence: +- Toggle proof: "With , repro produces . Reverting , repro produces ." +- Fix scope: +``` + +The "mechanism" field is the acid test. If you can't write the causal chain from cause to observable symptom as one paragraph, you don't yet understand the bug well enough to fix it. + +--- + +## Phase 7 — TDD Fix + +Red, green, refactor. No shortcuts. + +### 1. Red — failing-first test + +Write a test that fails *specifically because of this bug*. Requirements: + +- **Test name reads like a bug report.** `test_refinement_turn_returns_empty_content_when_anthropic_returns_401` is good. `test_bug_fix` is not. +- **Failure message clearly shows what the bug looks like.** If someone reads only the failure output, they understand what's broken. +- **Minimum infrastructure.** Don't spin up the whole server if a unit test against the right seam captures the mechanism. + +Run the test. Confirm it fails. Paste the failure output into the journal: + +```markdown +### Red phase () +Test: :: +Command: +Output: +``` + +``` +Confirms: the bug is reproducible at the test-harness level, not just the manual repro. +``` + +### 2. Green — minimum change + +Make the test pass with the **smallest change that fully fixes the observed mechanism**. + +If the diff is larger than ~30 lines and you aren't refactoring, something is wrong — either you're fixing more than the bug, or the root cause was deeper than you confirmed. Back to Phase 6. + +Signs you're over-fixing: +- Adding "just in case" null checks or try/except around other code +- Refactoring adjacent functions because "while I'm here" +- Adding new configuration options the bug didn't require +- Introducing new abstractions to "make this cleaner" + +Resist all of these. Fix the bug. Note the surrounding issues for follow-up. Move on. + +### 3. Refactor — ONLY AFTER GREEN + +Only cleanup directly related to the fix. Do not re-architect. + +If the code around the fix is rough, note it in the journal as a follow-up for the user; do not expand scope here. Refactoring during a bugfix is how one-line fixes turn into hundred-line diffs nobody can review. + +### 4. Regression — full suite green + +Run the full test suite for the affected package (not just the one new test). Existing tests must still pass. + +If they don't, your "fix" broke something else. Back to Phase 6 with the new failure as evidence — usually it means the mechanism you thought you fixed was load-bearing for some other code path you didn't know about, and the "broken" test is actually pointing at a better understanding of the system. + +### Update the journal + +```markdown +### Green phase () +Fix: — +Test: :: now passes +Full suite: +``` + +--- + +## The red-green discipline summary + +No red test → no proof the fix addresses the reported bug. Only proof it doesn't break tests that already existed. + +A test written *after* the fix might still pass with the fix reverted. If that's the case, the test doesn't lock the bug — it locks something else. Always verify the test fails without the fix and passes with it. The journal should show both outputs. diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/08-qa.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/08-qa.md new file mode 100644 index 000000000..d900e40e9 --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/08-qa.md @@ -0,0 +1,94 @@ +# Phase 8 — Manual QA by Actually Using It + +Tests cover cases you thought of. Real usage covers the ones you didn't. + +The single fastest way to ship a broken fix is to stop at "tests pass". Manual QA means interacting with the running system the way the user does, then comparing observed behavior to the original bug report. + +--- + +## Product-type playbook + +Pick the row that matches the product. Do what it says. Do not substitute. + +| Product type | QA means… | +|---|---| +| **CLI tool** | Open `tmux`, run the actual command end-to-end, capture output. Paste the session transcript into the journal. Include exit code, stdout, stderr, side-effect check (files created/modified). | +| **HTTP API** | Start the real server, hit endpoints with `curl` or `httpie`, inspect response status + body + headers. Hit the specific endpoint that reproduced the bug. If there's auth, use real auth. | +| **Browser-served web app** | **Drive a real browser via Playwright CLI.** See [tools/playwright-cli.md](../tools/playwright-cli.md). Navigate the exact page/flow that reproduced the bug. Capture screenshot + DOM + network evidence. **Do not substitute with curl** — browsers have state (cookies, localStorage, service workers, client-side JS, viewport-dependent CSS) that curl does not have. | +| **Agent / LLM pipeline** | Run the same user prompt that originally failed. Capture the full turn — tool calls, messages, usage counters. **Confirm non-zero usage** (zero usage = still failing silently, see silent-failure check below). | +| **Background worker / job queue** | Trigger the job through the normal entry point (API call, cron tick, message publish), tail the worker logs, observe completion state in the queue or DB. Don't just call the worker function directly — the trigger path matters. | +| **MCP server** | Invoke the tool via its actual client (Claude Desktop, Cursor, etc. if available) or `mcp-cli`, not just the HTTP probe endpoint. The MCP handshake itself is sometimes where bugs live. | +| **Native binary** | Re-run the exact command that crashed / misbehaved. If the input was a file, use the same file. If the bug was exploitable, confirm the exploit repro via pwntools (see [tools/pwntools.md](../tools/pwntools.md)). Capture exit code, signal if any, core dump if generated. | +| **Bundled-app binary** (Bun SEA, Node SEA, Electron, etc.) | Re-run the exact command. If the operation requires paid quota / blocked network, capture the **app's debug log** (`APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log`) which usually emits the assembled request before sending. See [methodology/partial-runtime-evidence.md](partial-runtime-evidence.md) for combining partial signals into a defensible verification. | +| **Long-running daemon** | Start fresh, let it run for the amount of time the bug originally took to manifest (not less), capture resource usage (memory, fd, cpu) throughout. Short-running QA misses resource leaks and cumulative state bugs. | + +--- + +## Journal format + +Every QA run goes in the journal under "Findings": + +```markdown +### Manual QA — () +- Scenario: +- Command: `` +- Observed output: +``` + +``` +- Expected output: +- Fix verified: yes / no / partial —
+``` + +If any QA step shows **partial or regressed behavior**, this is not "mostly done" — it's incomplete. Return to Phase 6. + +--- + +## The silent-failure check (always run) + +Regardless of product type, audit the fix against these silent-failure patterns. If the original bug was a silent failure, the same pattern may exist in adjacent code that you haven't tested yet. + +### Universal silent-failure signals + +- HTTP 2xx with empty or default body +- Response `ok: true` but a sub-field contains an error token (e.g. `stopReason: "error"`, `status: "failed"`) +- `usage.totalTokens === 0` on an LLM response +- Process exit code 0 but stderr contains an exception traceback +- Panic recovered and logged but ignored +- Goroutine / task / promise rejection with no top-level handler +- `try { ... } catch { /* swallowed */ }` or `except: pass` +- Success response shape but semantic field indicates failure (e.g. `error: null` actually being `error: "..."` with falsy check) +- Write returned success but read-back shows stale data +- Job marked complete but side-effect did not happen +- Cache hit path returned stale data and no refresh was triggered + +### Language-specific silent-failure signals + +Check the runtime reference for additional patterns: + +- [runtimes/python.md](../runtimes/python.md) — asyncio task exceptions, bare `except`, `logging.exception` that goes nowhere +- [runtimes/node.md](../runtimes/node.md) — unhandled promise rejections, `void` on async, swallowed `.catch(() => {})` +- [runtimes/rust.md](../runtimes/rust.md) — `.unwrap_or_default()`, `let _ = result`, error variants discarded +- [runtimes/go.md](../runtimes/go.md) — `if err != nil { return err }` that never reaches user output, recovered panics, buffered channels that block silently +- [runtimes/native-binary.md](../runtimes/native-binary.md) — ignored return codes from libc, missing `perror`, `alarm()` / signal masks +- [runtimes/bundled-js-binary.md](../runtimes/bundled-js-binary.md) — `process.env.X` baked at build time, dead code from tree-shaking failures, worker sub-bundles diverging from main bundle + +### What to do when you find another silent-failure spot + +Don't fix it. This is out of scope for the current bug. + +Note it in the journal under a "Follow-ups" section with: +- File:line +- Pattern matched +- Proposed fix sketch (one line) +- Risk level (what happens if left unfixed) + +Surface these to the user in the final message under "Next steps I didn't take". + +--- + +## The "fix verified" bar + +"Fix verified" means: the exact original failing scenario, re-run, now produces the correct output. Not a similar scenario. Not a unit test of the fix. The original scenario. + +If you can't re-run the original scenario (e.g. it required a specific data state that's gone), construct the closest equivalent and document the difference in the journal. Escalate to the user if the equivalent is materially different. diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/09-cleanup.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/09-cleanup.md new file mode 100644 index 000000000..ab87f1c8c --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/09-cleanup.md @@ -0,0 +1,164 @@ +# Phase 9 + 10 — Cleanup & Final Verification + +The working tree after the session must differ from before only by the real fix and its test. Anything else is a process failure. + +--- + +## Phase 9 — Cleanup & Revert + +### The walk + +Open the journal's "Artifacts to revert" list. Walk it top to bottom. Check each box only after the revert command succeeds and produces no error. + +### Standard revert operations + +Most sessions create some combination of these artifacts. The commands below are the defaults — your journal should have the exact commands for this session. + +```bash +# --- Temporary source edits (instrumentation statements, debug prints) --- +git checkout # reverts only that file +git diff # verify clean + +# --- tmux sessions --- +tmux kill-session -t +tmux ls # confirm gone + +# --- Temp fixtures / scratch scripts --- +rm -f /tmp/debug-*.* +ls /tmp/debug-*.* 2>/dev/null # confirm gone (ls returns non-zero when no match) + +# --- Background processes (debugger-attached runtimes) --- +pkill -f 'node --inspect' || true +pkill -f 'python -m pdb' || true +pkill -f 'debugpy' || true +pkill -f 'dlv' || true +pkill -f 'gdb' || true +pkill -f 'lldb' || true + +# --- Debug-relevant ports confirmed free --- +lsof -iTCP:9229 -sTCP:LISTEN -nP 2>/dev/null # Node inspector default +lsof -iTCP:5678 -sTCP:LISTEN -nP 2>/dev/null # debugpy default +lsof -iTCP:2345 -sTCP:LISTEN -nP 2>/dev/null # dlv default +lsof -iTCP:9999 -sTCP:LISTEN -nP 2>/dev/null # pwndbg/gdb-server default + +# --- Env var overrides in current shell --- +unset DEBUG_OVERRIDE_FOO +unset PYTHONBREAKPOINT +unset RUST_LOG +unset DEBUG + +# --- Ghidra scratch projects (if created just for this session) --- +# rm -rf ~/ghidra-projects/debug-scratch + +# --- Core dumps from debugging (if any) --- +rm -f ./core ./core.* ~/core.* + +# --- Playwright trace files --- +rm -rf playwright-report/ test-results/ +``` + +### The verify command + +This is the single most important check of the whole skill: + +```bash +git status +git diff --stat +``` + +The diff must contain **only**: + +1. The real fix. +2. The new failing-first test. +3. Nothing else. + +### Detector checklist — scan the diff for these + +If `git status` shows any untracked debug file, or `git diff` shows any of the patterns below, **you are not done**. Clean it. + +| Pattern | Usually means | +|---|---| +| `debugger;` | Node debug statement left behind | +| `breakpoint()` | Python debug statement left behind | +| `dbg!(...)` | Rust debug macro left behind | +| `fmt.Println("DEBUG: ...")` | Go ad-hoc print | +| `console.log("[DEBUG]` | Node ad-hoc log | +| `print(f"DEBUG: ` | Python ad-hoc print | +| `// TODO DEBUG`, `// HACK`, `// XXX` | Stale debug marker | +| `// -DEBUG` | Session-specific marker from this skill's edits | +| Commented-out code blocks near the fix | Dead code from trial fixes | +| Reordered imports or formatting in unrelated files | Drift from your editor's autoformat during the session | + +### Remove the journal + +Only once the git check is clean: + +```bash +rm .debug-journal.md +sed -i.bak '/^\.debug-journal\.md$/d' .git/info/exclude && rm -f .git/info/exclude.bak +``` + +The journal is not part of the fix; it doesn't belong in the commit or in the git exclude list. + +--- + +## Phase 10 — Final Verification + +Last gate before reporting done. All four gates must be true, and all four must have **evidence in your final message** to the user. Passing a gate without evidence is the same as failing it. + +### The four gates + +1. **Red→green toggle confirmed** — show the failing test output from before the fix and passing output after. Both outputs visible in the reply or the journal. + +2. **Full test suite green** — show the suite's final pass line (e.g. `42 passed in 3.14s`). Not just the new test. + +3. **Manual QA reproduced the fix** — show the command or scenario that originally failed and its now-correct output. Verbatim, not paraphrased. + +4. **Working tree clean of debug artifacts** — show `git diff --stat` output containing only fix + test, plus `git status` clean of untracked debug files. + +If any of the four lacks evidence, you have not finished — return to the appropriate phase. + +### Final message template + +Keep it short. Evidence-dense. The user should be able to skim it in 30 seconds. + +```markdown +Fixed. + +**Root cause**: +**Fix**: `` — +**Test**: `::` — red without fix, green with fix +**QA**: + +Diff: +``` + +``` + +**Next steps I didn't take** (awaiting your decision): +- +- +``` + +### Example (from a real session) + +```markdown +Fixed. + +**Root cause**: pi-mono Agent's `model.baseUrl` was hardcoded to `api.anthropic.com`, so the `ANTHROPIC_BASE_URL` env var was silently ignored. The proxy API key was rejected by the real Anthropic API with 401, but pi-mono packaged the error into the assistant message's `errorMessage` field instead of throwing, so the route's try/catch never fired and the client received HTTP 200 with empty content. + +**Fix**: `core/pi-bridge/modelResolver.ts:117` — override baseUrl +**Test**: `__tests__/core/modelResolver.test.ts::resolves_env_override` — red without fix, green with fix +**QA**: `curl -X POST /api/refinement/chat` with proxy env set, observed non-zero usage and non-empty content + +Diff: +``` + core/pi-bridge/modelResolver.ts | 3 +++ + __tests__/core/modelResolver.test.ts | 42 ++++++++++++++++++++++ + 2 files changed, 45 insertions(+) +``` + +**Next steps I didn't take** (awaiting your decision): +- pi-mono itself silently swallows LLM errors into `errorMessage`; adding a throw-on-error wrapper at our orchestrator layer would surface these upstream +- Same silent-failure pattern exists in the planning route — likely the same fix applies +``` diff --git a/packages/omo-codex/plugin/skills/debugging/references/methodology/partial-runtime-evidence.md b/packages/omo-codex/plugin/skills/debugging/references/methodology/partial-runtime-evidence.md new file mode 100644 index 000000000..986a3220c --- /dev/null +++ b/packages/omo-codex/plugin/skills/debugging/references/methodology/partial-runtime-evidence.md @@ -0,0 +1,229 @@ +# Partial Runtime Evidence — When You Cannot Execute the Real Operation + +Read this when **runtime truth beats code reading** is in conflict with **you cannot run the actual operation**. + +The skill's first invariant is "runtime state is the only source of truth." But sometimes the only state you can produce is a *partial* observation — the real call requires paid credits, a hardware device you don't have, network access through a corporate proxy, a production secret, or a customer dataset. + +**Partial runtime evidence is still runtime evidence.** This reference tells you which partial signals to harvest and how to combine them so the conclusion is defensible. + +--- + +## When this applies + +Use this reference when ALL are true: + +1. The bug or extraction question requires runtime confirmation (per skill invariant #1). +2. You attempted the obvious "just run it" path and it failed for reasons unrelated to the bug: + - 401/402/403 from a paid API + - "device not found" / "permission denied" / SIP block + - Production-only credentials + - Network isolation (air-gapped, behind VPN you don't have) + - Time-of-day or quota limits +3. **Mocking the entire system** would defeat the verification — you specifically need evidence about how the *real* code behaves, not a stub. + +If only #1 and #2 are true and you can mock cleanly, just mock and proceed. This file is for cases where mocking would invalidate the answer. + +--- + +## The hierarchy of partial evidence (strongest first) + +When you cannot capture the full outbound payload + full response, capture as much as possible from this list. **Evidence further down the list has more inference; evidence higher up is closer to ground truth.** + +### Tier 1 — Pre-send / post-receive logs (best partial evidence) + +The system you're investigating builds a request, then sends it. If the build step logs the assembled request **before** transmission, that log is ground truth for everything except the wire-level bytes (TLS, headers added by HTTP library, etc.). + +```bash +# Maximize debug logging +APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target -x "minimal valid input" 2>&1 | head -200 +``` + +Look for log lines like: +- `Building request: model=X, params={...}` +- `[provider] payload: {...}` +- `Sending to : ` + +**Strength**: 95% of ground truth. Missing only wire-level transformations. + +### Tier 2 — Local interception via proxy / shim + +Run the real binary against a local proxy that records and (optionally) returns a canned response. + +```bash +# mitmproxy approach +mitmproxy --listen-host 127.0.0.1 --listen-port 8888 --mode regular & +HTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ... +# Now mitmproxy logs the actual TLS-decrypted request +``` + +```bash +# DYLD_INSERT_LIBRARIES / LD_PRELOAD shim approach +# Wrap the network call to log payload, return a fake 200 +# See pwntools.md for shim examples +``` + +**Strength**: Wire-level ground truth, but requires the target to honor your proxy / preload. + +### Tier 3 — Static extraction × runtime fingerprint cross-check + +When you cannot send a request at all, you can still cross-check static analysis with whatever the binary does that *doesn't* require the real call: + +- The binary builds the request — even if sending fails, the build step ran. Trace it (Tier 1). +- The binary writes a state file or cache — read it. +- The binary emits version-specific User-Agent strings; verify they match your static extraction. +- The binary's `--help` or `--version` output reveals build metadata; verify model lists / feature flags. + +**Strength**: Disjoint evidence sources confirming the same fact. Two independent partial signals that agree are nearly as strong as one full observation. + +### Tier 4 — Contrastive runtime under different inputs + +If you can run with input variant A but not B, run A and reason about B from code: + +```bash +# A: minimal trial input — works for free tier +./target --action=read --resource=local-file +# B: full inference call — paid tier required, blocked +# But the request-building code is shared between A and B! +# Capture A's logs, then inspect the code path for B and verify only the model/endpoint diff. +``` + +**Strength**: Confirms shared code paths; remaining gap is only the difference between A and B. + +### Tier 5 — Vendor-published API logs / dashboard + +If the operation succeeded earlier (before quota ran out, before access was revoked), the vendor's dashboard / audit log may show the request. Lower fidelity but still observed behavior. + +**Strength**: Real wire data, but often summarized — token counts, status codes, no payload bodies. + +### Tier 6 — Pure code reading with peer review + +If literally none of the above is available, read the code carefully and submit it to **one Oracle for skeptical review** (see "Verification Oracle" below). This is the weakest tier and you must explicitly mark conclusions as "unverified" in the journal. + +--- + +## How to combine partial signals + +A defensible conclusion **prefers two independent signals from different tiers**, with one exception: a complete Tier 2 wire-level capture is wire-level ground truth and can stand alone for request-shape claims (because the wire bytes are exactly what the remote received). For *behavioral* claims (what the system does next, what state it stores, what side effects it produces), still combine with another signal. + +| Available evidence | Defensibility | +|---|---| +| Tier 1 + Tier 1 (same log, different lines) | weak — single source | +| Tier 1 + Tier 2 (debug log + proxy capture) | **strong** — independent confirmation | +| Tier 1 + Tier 3 (debug log + version output cross-check) | **strong** — disjoint sources | +| Tier 2 alone (full proxy capture) | strong **for request-shape claims only** — stands alone for "what bytes were sent". Add a second signal for response-handling or state claims. | +| Tier 3 + Tier 4 (cross-check + contrastive run) | medium — both partial | +| Tier 6 alone (code reading only) | **insufficient** — escalate or mark unverified | + +Record in the journal: + +```markdown +## Partial runtime evidence +### Question being verified + + +### Available signals +- Tier 1: debug log /tmp/trace.log line 47-49 shows `effort: "high"` ✓ +- Tier 3: static extraction of m5T() function returns "high" for smart mode ✓ +- Tier 6: code path verified by reading prompt-builder.js ✓ + +### Independence assessment +Tier 1 and Tier 3 are independent — the log was emitted by a different +code path than m5T() and would diverge if the static reading were wrong. + +### Conclusion +VERIFIED via Tier 1 + Tier 3 agreement. No need to escalate. +``` + +If you cannot achieve a complete Tier 2 capture **or** two independent non-Tier-6 signals from the table above, **write an explicit note in the deliverable**: + +> ⚠️ Partial-evidence finding. The full outbound payload could not be captured because [reason]. The conclusion rests on: +> - [signal A — tier and source] +> - [signal B — tier and source] +> A future verification should attempt [the missing tier] when [condition]. + +--- + +## Verification Oracle pattern (for non-debug tasks) + +The skill's main Oracle Triple (`04-oracle-triple.md`) is for **stuck debugging** — 2 failed rounds, mental box, three orthogonal framings to break out. + +For tasks where the deliverable is an **artifact, not a bug fix** (reverse engineering, extraction, audit, compliance documentation), use a different pattern: **single Oracle, late, skeptical, with the deliverable in hand**. + +### When to invoke + +- Right before declaring an extraction/audit task "done" +- After every significant revision of the deliverable (not after every small edit) +- Maximum 3-4 iterations before escalating to user + +### Pattern + +``` +task(subagent_type="oracle", load_skills=[], run_in_background=false, + prompt=""" +SKEPTICAL FINAL VERIFICATION — be critical, look for reasons the task is incomplete or wrong. + +## Original task + + +## What I produced + + +## Specific claims to verify + + +## Where to look + + +## Your job +1. Read the deliverables. +2. Spot-check each claim against the source/evidence the deliverable cites. +3. Identify any unsubstantiated claims, missing pieces, or factual errors. +4. End with PASS / FAIL / PARTIAL with specific gaps. +Be skeptical. Don't rubber-stamp. +""") +``` + +### Why this differs from the Oracle Triple + +| | Oracle Triple (debug) | Verification Oracle (artifact) | +|---|---|---| +| Trigger | 2 failed hypothesis rounds | About to declare "done" | +| Count | 3 in parallel, orthogonal framings | 1 sequential, focused review | +| Goal | Break out of mental box | Catch unsubstantiated claims | +| Tone of prompt | Brainstorm wide alternatives | Skeptical audit | +| Iteration | Reset hypothesis set after | Fix gaps, re-invoke until PASS | + +### Don't conflate them + +If you're stuck debugging, do the Triple. If you have a deliverable and need it audited, do the Verification Oracle. Doing the Triple on a finished extraction will return three diverging "what if you tried…" tangents that are not what you need. Doing the Verification Oracle on a stuck debugging session will return a polite "the evidence is incomplete" that you already knew. + +--- + +## Common partial-evidence anti-patterns + +| Anti-pattern | Why it fails | Replacement | +|---|---|---| +| "It looks right in the code, so it works" | Tier 6 alone, unverified | Add at least one Tier 1-3 signal | +| "I ran it once, didn't error, so it's correct" | Absence of error ≠ presence of correctness | Capture the actual output and verify content | +| "The mock returns the value I wrote, so the code is fine" | Tautology — mock loops back your assumption | Use Tier 2 (proxy) instead, or cross-check with Tier 3 | +| "The vendor's dashboard shows my call worked" | Dashboard often only shows status code, not behavior | Combine with Tier 1 if available | +| "I'll trust the most-recent stack overflow answer" | Code from a different version / context | Verify against the actual binary you have | + +--- + +## Cleanup additions for partial-evidence work + +```bash +# Proxy artifacts +pkill -f mitmproxy 2>/dev/null +rm -f ~/.mitmproxy/cache_* 2>/dev/null + +# Debug log files +rm -f /tmp/trace.log /tmp/*-debug-trace.log + +# DYLD_INSERT / LD_PRELOAD shim libraries +rm -f /tmp/*.dylib /tmp/*.so + +# Verify env vars set in your shell are not persisted +unset HTTPS_PROXY APP_DEBUG APP_LOG_LEVEL APP_LOG_FILE 2>/dev/null +```