From 73e8988f317d6f4df0b6d0abaf7a709349ddebfd Mon Sep 17 00:00:00 2001 From: YeonGyu-Kim Date: Wed, 27 May 2026 13:26:10 +0900 Subject: [PATCH] feat(omo-codex): enumerate 4 manual-QA channels + forbid tests-alone verification in ultrawork/ultragoal Promote the four real-usage QA channels (HTTP call, tmux, Browser use, Computer use) into a dedicated top-level `# Manual-QA channels` section in both the Codex ultrawork directive and the ultragoal skill. Each channel ships with the concrete invocation (`curl -i`, `tmux new-session` + `send-keys` + `capture-pane`, Playwright / puppeteer / Chromium, OS-level GUI automation) and the artifact to capture. The Goal section now declares `TESTS ALONE NEVER PROVE DONE` and requires every criterion to build a fresh real-usage scenario and run it through one of the four channels every time. Auxiliary surfaces (CLI stdout / DB diff / parsed config dump) are explicitly demoted: they only satisfy genuinely CLI- or data-shaped criteria, never user-facing behavior. Bootstrap criterion item 2 and execution step 4 used to repeat the same surface enumeration; they now collapse onto the new channel table (single source of truth, less drift). EXECUTE-AS-SCENARIO in the ultragoal skill follows the same compression. The previous CLEANUP (paired teardown + receipt) + leftover-state stop rule from the prior commit are preserved. Regression tests in `components/ultrawork/hooks/ultrawork-hooks.test.mjs` pin: the `# Manual-QA channels` heading, all four channel labels (`HTTP call`, `tmux`, `Browser use`, `Computer use`), `TESTS ALONE NEVER PROVE DONE`, `every criterion needs its own real-usage scenario`, and `every time`. README + CHANGELOGs document the restructure and the new size (11,005 chars / 232 lines). All 11 ultrawork hook tests + 7 aggregate tests pass. Installed locally via `runCodexInstaller` into `~/.codex/plugins/cache/code-yeongyu-codex-plugins/omo/0.1.0/`; cache hooks emit the new directive end-to-end and the identifier-like `ulw_helper.ts` keyword-boundary check still suppresses. --- .../plugin/components/ultragoal/CHANGELOG.md | 3 +- .../ultragoal/skills/ultragoal/SKILL.md | 23 +++++-- .../plugin/components/ultrawork/CHANGELOG.md | 17 +++-- .../plugin/components/ultrawork/README.md | 7 +- .../ultrawork/hooks/ultrawork-detector.py | 67 ++++++++++++------- .../ultrawork/hooks/ultrawork-hooks.test.mjs | 34 +++++++++- .../plugin/skills/ultragoal/SKILL.md | 23 +++++-- 7 files changed, 128 insertions(+), 46 deletions(-) diff --git a/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md b/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md index 688781ba0..72a3e0986 100644 --- a/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md +++ b/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md @@ -3,4 +3,5 @@ ## [0.1.0] - unreleased - Initial scaffold of codex-ultragoal plugin. -- Per-Criterion Cycle: `EXECUTE` is now **EXECUTE-AS-SCENARIO** — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). Inserted a new **CLEAN (PAIRED, NEVER SKIP)** step that tears down every QA-spawned process / `tmux` session / browser context / container / port / temp dir before recording evidence; the cleanup receipt is embedded in the `--evidence` string. Missing receipt → record BLOCKED, not PASS. Added Constraint #13 and a Stop Rule for leftover state. +- Per-Criterion Cycle: `EXECUTE` is now **EXECUTE-AS-SCENARIO** — the agent must run the Manual-QA channel scenario the criterion named (HTTP call / tmux / browser use / computer use; see new `## Manual-QA channels` section). Inserted a new **CLEAN (PAIRED, NEVER SKIP)** step that tears down every QA-spawned process / `tmux` session / browser context / container / port / temp dir before recording evidence; the cleanup receipt is embedded in the `--evidence` string. Missing receipt → record BLOCKED, not PASS. Added Constraint #13 and a Stop Rule for leftover state. +- New top-level **`## Manual-QA channels`** section explicitly enumerates the four channels (HTTP call, tmux, Browser use, Computer use) with concrete commands and required artifacts. Goal section now declares **TESTS ALONE NEVER PROVE DONE**: a green test suite is supporting evidence, never completion proof. Criterion-refinement step 2 requires each criterion to name its channel up front. diff --git a/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md b/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md index eb8d70e03..47e8784a6 100644 --- a/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md +++ b/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md @@ -9,9 +9,20 @@ Use GPT-5.x style: outcome-first, evidence-bound, atomic decisions, no nested br ## Goal Deliver every goal in `.omo/ultragoal/goals.json` end-to-end. -Prove EVERY success criterion with captured observable evidence from the real surface. +Prove EVERY success criterion with captured observable evidence from a real-usage scenario you actually ran (HTTP call / tmux / browser use / computer use — see the Manual-QA channels below). +TESTS ALONE NEVER PROVE DONE. A green test suite is supporting evidence, not completion proof. Audit each pass, fail, block, steering change, and checkpoint in `.omo/ultragoal/ledger.jsonl`. +## Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT) +For every criterion, build a real-usage scenario through ONE of these four channels and run it yourself before recording PASS. The full test suite being green is NEVER verification on its own. + +1. **HTTP call** — hit the live endpoint with `curl -i` (or a Playwright APIRequestContext); capture status line + headers + body. +2. **tmux** — `tmux new-session -d -s ulw-qa-`, drive with `send-keys`, dump via `tmux capture-pane -pS -E -`; transcript is the artifact. +3. **Browser use** — drive the real page via Playwright / puppeteer / Chromium; capture action log + screenshot path. +4. **Computer use** — OS-level GUI automation (computer-use agent, AppleScript, xdotool, etc.) against the running app; capture action log + screenshot. + +Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config dump) satisfy CLI- or data-shaped criteria but NEVER replace a channel scenario for user-facing behavior. `--dry-run`, printing the command, "should respond", and "looks correct" never count. + ## Artifacts - `.omo/ultragoal/brief.md`: original brief and durable constraints. - `.omo/ultragoal/goals.json`: goals with embedded `successCriteria` per goal. @@ -34,12 +45,12 @@ Write state through the CLI path. Do not hand-edit state files. ### 2. Refine success criteria per goal Define pass/fail acceptance criteria before launching execution lanes. Include the command, artifact, or manual check that will prove success. Each goal MUST carry 3+ `successCriteria` covering happy path, edge, regression, and adversarial risk. -For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, and stop condition. +For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, stop condition, and the Manual-QA channel (HTTP call / tmux / browser use / computer use) that will exercise it. Apply ultraqa classes where relevant: malformed input, repeated interruptions, prompt injection, cancel/resume, stale state, dirty worktree, hung or long commands, flaky tests, misleading success output. -Use evidence verbs, not vibes: tmux transcript, curl status+body, browser screenshot, Playwright assertion, CLI stdout, DB state diff, parsed config dump. -"Tests pass" is supporting signal, not completion proof. +Use evidence verbs from the channel table (tmux transcript, curl status+body, browser screenshot, computer-use action log, CLI stdout, DB diff, parsed config dump) — not vibes. +"Tests pass" is supporting signal, NEVER completion proof. Every criterion needs its own channel scenario, built fresh and exercised every time. Record manual QA notes when behavior is user-visible. -Revise any criterion that lacks observable `expectedEvidence` before execution. +Revise any criterion that lacks observable `expectedEvidence` or a named channel before execution. ### 3. Inspect state Run `omo ultragoal status --json`. @@ -64,7 +75,7 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a ### Per-Criterion Cycle 1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds. 2. Register atomic todos: `path: for - verify by `. -3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count. +3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY run the Manual-QA channel scenario the criterion named (HTTP call / tmux / browser use / computer use — see the channel table above). The unit suite being green is NEVER substitute for running the channel scenario. 4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump. 5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS. 6. RECORD exactly one result: diff --git a/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md b/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md index 6776a832a..0153a0252 100644 --- a/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md +++ b/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md @@ -2,11 +2,18 @@ ## Unreleased -- Execution loop now mandates **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count. -- New paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact (server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars) with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress. -- New Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done. -- Regression tests in `hooks/ultrawork-hooks.test.mjs` pin the SURFACE-AS-SCENARIO + CLEANUP mandates so they cannot be silently regressed. -- Directive size: 10,037 chars across 213 lines. +- New top-level **`# Manual-QA channels`** section explicitly enumerates the four real-usage channels the agent MUST verify through: (1) HTTP call, (2) tmux, (3) Browser use, (4) Computer use — each with concrete commands and the artifact to capture. Auxiliary surfaces (CLI stdout / DB diff / parsed config dump) only count for genuinely CLI- or data-shaped criteria. +- Goal section now shouts **TESTS ALONE NEVER PROVE DONE**: a green test suite is supporting evidence, never completion proof. Every criterion needs its own real-usage scenario, built fresh and run through one of the four channels, every time. +- Bootstrap criterion item 2 and execution step 4 collapse onto the new channel table to remove triple-enumeration of the same surfaces (single source of truth, less drift). +- Execution loop step 4 (**SURFACE-AS-SCENARIO**) runs the chosen channel scenario; step 5 (**CLEANUP, PAIRED**) tears down server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars and records a one-line receipt. Missing receipt → criterion stays in_progress. Leftover state from QA = NOT done (Stop rule). +- Regression tests in `hooks/ultrawork-hooks.test.mjs` now pin: the four channel labels (`HTTP call`, `tmux`, `Browser use`, `Computer use`), `TESTS ALONE NEVER PROVE DONE`, `every criterion needs its own real-usage scenario`, the `# Manual-QA channels` heading, plus SURFACE-AS-SCENARIO + CLEANUP + leftover-state stop rule. +- Directive size: 11,005 chars across 232 lines. + +### Pre-cleanup unreleased entries (folded above) + +- Execution loop mandated **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count. +- Paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress. +- Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done. ## 0.1.0 — 2026-05-23 diff --git a/packages/omo-codex/plugin/components/ultrawork/README.md b/packages/omo-codex/plugin/components/ultrawork/README.md index 76dc8a0f7..5c13cce0e 100644 --- a/packages/omo-codex/plugin/components/ultrawork/README.md +++ b/packages/omo-codex/plugin/components/ultrawork/README.md @@ -6,13 +6,14 @@ Codex plugin that injects a compact orchestration directive (the **ultrawork** p | Mandate | Behavior | |---|---| -| Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario's PASS condition is **observable evidence from the real surface** (`tmux` transcript, `curl` status+body, browser screenshot, Playwright assertion, computer-use action log, CLI stdout, parsed config dump, DB state diff). "Tests pass" alone is not evidence. | -| Manual-QA-as-scenario + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) forces the agent to *actually invoke* the surface end-to-end with concrete commands keyed by surface (HTTP → `curl -i`, TUI → `tmux new-session` + `send-keys` + `capture-pane`, GUI → computer-use / Playwright, CLI, DB diff). Step 5 (**CLEANUP, PAIRED**) requires teardown of every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. | +| Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario MUST name which **Manual-QA channel** it will use. "Tests pass" is supporting signal, NEVER completion proof. | +| Manual-QA channels (TESTS ALONE NEVER PROVE DONE) | A dedicated top-level section enumerates the **four** channels you can use to verify a criterion in reality: **(1) HTTP call** (`curl -i` / Playwright APIRequestContext), **(2) tmux** (`tmux new-session` + `send-keys` + `capture-pane`), **(3) Browser use** (Playwright / puppeteer / Chromium driving the real page), **(4) Computer use** (OS-level GUI automation against the running app). Every criterion picks one channel, builds a real-usage scenario, runs it, and captures the artifact — every time. Aux surfaces (CLI stdout / DB diff / parsed config) only count for genuinely CLI- or data-shaped criteria. | +| Surface + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) runs the chosen channel scenario end-to-end. Step 5 (**CLEANUP, PAIRED**) tears down every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. | | Durable /tmp notepad | `mktemp -t ulw-$(date +%Y%m%d-%H%M%S).XXXXXX.md` with sections `Plan`, `Success criteria + QA scenarios`, `Now`, `Todo`, `Findings`, `Learnings`. **Append**, never rewrite. | | Obsessive atomic todos | Every action — even one-line edits, `ls`, single test runs — becomes a todo. Format: `path: for — verify by `. One in_progress at a time, mark completed immediately. | | GPT-5.2 xhigh verification gate | Triggered automatically on user-requested rigor, 3+ files, 20+ turns, 30+ minutes, or refactor/migration/perf/security work. Use the bundled `codex-ultrawork-reviewer` agent role when available. Reviewer verdict is **binding** — no "false positive", no minimising, no arguing. Loop until **unconditional** approval. "Looks good but…" = REJECTION. | -The directive is currently 10,037 chars / 213 lines and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules). +The directive is currently 11,005 chars / 232 lines and follows the GPT-5.5 prompting structure (Role / Goal / Manual-QA channels / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules). ## Install (via this marketplace) diff --git a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py index ee63838ba..413dd04d1 100755 --- a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py +++ b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py @@ -35,9 +35,37 @@ narration. # Goal Deliver EXACTLY what the user asked, end-to-end working, proven by -(a) a test written test-first that went RED→GREEN and (b) manual QA -from the real surface with captured observable evidence. BOTH gates, -every change, no exceptions. +(a) a test written test-first that went RED→GREEN and (b) a manual-QA +scenario you actually run against the real surface (HTTP call / tmux / +browser use / computer use — see the channel table below) with the +artifact captured. Both gates, every change, no exceptions. +TESTS ALONE NEVER PROVE DONE. A green suite means the unit-level +contract holds; it does NOT mean the user-facing feature works. Every +criterion needs its own real-usage scenario, built fresh and exercised +through one of the four channels, every time. + +# Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT) +For every criterion, build a real-usage scenario through ONE of these +four channels and run it yourself before declaring the criterion done. +The full test suite being green is NEVER verification on its own. + + 1. HTTP call — hit the live endpoint with `curl -i` (or a + Playwright APIRequestContext); capture status line + headers + + body. + 2. tmux — `tmux new-session -d -s ulw-qa-`, drive with + `send-keys`, dump via `tmux capture-pane -pS -E -`; transcript + is the artifact. + 3. Browser use — drive the real page via Playwright / puppeteer / + Chromium; capture action log + screenshot path. + 4. Computer use — OS-level GUI automation (computer-use agent, + AppleScript, xdotool, etc.) against the running app; capture + action log + screenshot. + +Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config +dump) are valid evidence when the criterion is genuinely CLI- or +data-shaped, but they do NOT replace a channel scenario for any +user-facing behavior. `--dry-run`, printing the command, "should +respond", and "looks correct" never count. # Bootstrap (DO ALL THREE BEFORE ANY OTHER WORK — NO SKIPPING) @@ -58,15 +86,13 @@ The criteria MUST list, upfront: 1. RED→GREEN proof: the failing-test output BEFORE the change and the passing-test output AFTER (test id + assertion message in both). Tests added AFTER the green code do NOT satisfy this. - 2. Manual-QA-as-scenario artifact — you (the agent) actually - INVOKE the real surface, not just claim it would work: HTTP via - `curl` (status + body), terminal / TUI via `tmux` (new session, - send-keys, `capture-pane -p` transcript), GUI via computer-use - or Playwright (action log + screenshot), CLI stdout, DB state - diff. "Should respond" / "looks correct" is NOT QA. - Tests are the FLOOR (required, never sufficient); the surface - artifact is the CEILING (also required). "tests pass" alone is NOT - done. + 2. Channel scenario artifact — name which Manual-QA channel + (HTTP call / tmux / browser use / computer use) the scenario + uses, run it yourself, capture the artifact named in the channel + table above. + Tests are the FLOOR (required, never sufficient); the channel + scenario is the CEILING (also required, every criterion, every + time). "tests pass" alone is NEVER done. These scenarios are the contract. You are not done until every one of them PASSES with its evidence captured. @@ -128,18 +154,11 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces: Re-run the test. Capture GREEN output. If GREEN required more than ~20 lines, your test was too coarse — split it. 4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS): - ACTUALLY invoke the surface end-to-end as the user would. Concrete - moves keyed by surface (use the one the criterion names): - • HTTP → `curl -i` the running server; capture status + body. - • Terminal / TUI / interactive CLI → `tmux new-session -d -s - ulw-qa-`, drive with `send-keys`, dump via - `tmux capture-pane -pS -E -`. - • GUI / web → computer-use / Playwright; capture action log + - screenshot path. - • Pure CLI → run it; capture stdout + exit code. - • DB / state-mutating → before/after diff. - Paste the artifact path into the notepad. `--dry-run`, printing the - command, or "looks correct" does NOT count. + Run the Manual-QA channel scenario the criterion named (HTTP + call / tmux / browser use / computer use; see the channel table at + the top). Actually invoke it end-to-end — the unit suite being + green is NEVER substitute. Paste the artifact path into the + notepad. 5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA spawned in step 4 MUST be torn down before this step completes: server PIDs (`kill `; verify `kill -0` fails), `tmux` sessions diff --git a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs index 3f39e520a..04180ecf6 100644 --- a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs +++ b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs @@ -133,7 +133,39 @@ test("#given ultrawork prompt #when detector runs #then directive mandates manua assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/); assert.match(result.stdout, /curl -i/); assert.match(result.stdout, /tmux new-session/); - assert.match(result.stdout, /computer-use \/ Playwright/); +}); + +test("#given ultrawork prompt #when detector runs #then directive enumerates 4 manual-QA channels explicitly", async () => { + const payload = JSON.stringify({ + hook_event_name: "UserPromptSubmit", + prompt: "please ultrawork", + }); + + const result = await runPython(detectorPath, payload); + + assert.equal(result.code, 0); + assert.equal(result.stderr, ""); + assert.match(result.stdout, /# Manual-QA channels/); + assert.match(result.stdout, /PICK ONE PER CRITERION \u2014 ACTUALLY RUN IT/); + assert.match(result.stdout, /1\. HTTP call/); + assert.match(result.stdout, /2\. tmux/); + assert.match(result.stdout, /3\. Browser use/); + assert.match(result.stdout, /4\. Computer use/); +}); + +test("#given ultrawork prompt #when detector runs #then directive forbids tests-alone verification", async () => { + const payload = JSON.stringify({ + hook_event_name: "UserPromptSubmit", + prompt: "please ultrawork", + }); + + const result = await runPython(detectorPath, payload); + + assert.equal(result.code, 0); + assert.equal(result.stderr, ""); + assert.match(result.stdout, /TESTS ALONE NEVER PROVE DONE/); + assert.match(result.stdout, /Every[\s\n]+criterion needs its own real-usage scenario/); + assert.match(result.stdout, /every time/); }); test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => { diff --git a/packages/omo-codex/plugin/skills/ultragoal/SKILL.md b/packages/omo-codex/plugin/skills/ultragoal/SKILL.md index eb8d70e03..47e8784a6 100644 --- a/packages/omo-codex/plugin/skills/ultragoal/SKILL.md +++ b/packages/omo-codex/plugin/skills/ultragoal/SKILL.md @@ -9,9 +9,20 @@ Use GPT-5.x style: outcome-first, evidence-bound, atomic decisions, no nested br ## Goal Deliver every goal in `.omo/ultragoal/goals.json` end-to-end. -Prove EVERY success criterion with captured observable evidence from the real surface. +Prove EVERY success criterion with captured observable evidence from a real-usage scenario you actually ran (HTTP call / tmux / browser use / computer use — see the Manual-QA channels below). +TESTS ALONE NEVER PROVE DONE. A green test suite is supporting evidence, not completion proof. Audit each pass, fail, block, steering change, and checkpoint in `.omo/ultragoal/ledger.jsonl`. +## Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT) +For every criterion, build a real-usage scenario through ONE of these four channels and run it yourself before recording PASS. The full test suite being green is NEVER verification on its own. + +1. **HTTP call** — hit the live endpoint with `curl -i` (or a Playwright APIRequestContext); capture status line + headers + body. +2. **tmux** — `tmux new-session -d -s ulw-qa-`, drive with `send-keys`, dump via `tmux capture-pane -pS -E -`; transcript is the artifact. +3. **Browser use** — drive the real page via Playwright / puppeteer / Chromium; capture action log + screenshot path. +4. **Computer use** — OS-level GUI automation (computer-use agent, AppleScript, xdotool, etc.) against the running app; capture action log + screenshot. + +Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config dump) satisfy CLI- or data-shaped criteria but NEVER replace a channel scenario for user-facing behavior. `--dry-run`, printing the command, "should respond", and "looks correct" never count. + ## Artifacts - `.omo/ultragoal/brief.md`: original brief and durable constraints. - `.omo/ultragoal/goals.json`: goals with embedded `successCriteria` per goal. @@ -34,12 +45,12 @@ Write state through the CLI path. Do not hand-edit state files. ### 2. Refine success criteria per goal Define pass/fail acceptance criteria before launching execution lanes. Include the command, artifact, or manual check that will prove success. Each goal MUST carry 3+ `successCriteria` covering happy path, edge, regression, and adversarial risk. -For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, and stop condition. +For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, stop condition, and the Manual-QA channel (HTTP call / tmux / browser use / computer use) that will exercise it. Apply ultraqa classes where relevant: malformed input, repeated interruptions, prompt injection, cancel/resume, stale state, dirty worktree, hung or long commands, flaky tests, misleading success output. -Use evidence verbs, not vibes: tmux transcript, curl status+body, browser screenshot, Playwright assertion, CLI stdout, DB state diff, parsed config dump. -"Tests pass" is supporting signal, not completion proof. +Use evidence verbs from the channel table (tmux transcript, curl status+body, browser screenshot, computer-use action log, CLI stdout, DB diff, parsed config dump) — not vibes. +"Tests pass" is supporting signal, NEVER completion proof. Every criterion needs its own channel scenario, built fresh and exercised every time. Record manual QA notes when behavior is user-visible. -Revise any criterion that lacks observable `expectedEvidence` before execution. +Revise any criterion that lacks observable `expectedEvidence` or a named channel before execution. ### 3. Inspect state Run `omo ultragoal status --json`. @@ -64,7 +75,7 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a ### Per-Criterion Cycle 1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds. 2. Register atomic todos: `path: for - verify by `. -3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count. +3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY run the Manual-QA channel scenario the criterion named (HTTP call / tmux / browser use / computer use — see the channel table above). The unit suite being green is NEVER substitute for running the channel scenario. 4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump. 5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS. 6. RECORD exactly one result: