feat(omo-codex): mandate manual-QA-as-scenario + paired cleanup in ultrawork/ultragoal

Both the Codex ultrawork directive and the ultragoal skill now force the agent to actually invoke the real user-facing surface (HTTP via `curl -i`, terminal/TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff) instead of treating evidence as a free-form artifact list. A paired CLEANUP step requires teardown of every QA-spawned process, tmux session, browser context, container, bound port, temp file/dir, and QA-only env var, with a one-line cleanup receipt recorded next to the artifact path (ultrawork) or embedded in the `--evidence` string (ultragoal). Missing receipt keeps the criterion in_progress / records BLOCKED. New Stop rule: leftover state from QA means NOT done. Regression tests in `components/ultrawork/hooks/ultrawork-hooks.test.mjs` pin SURFACE-AS-SCENARIO, the concrete `curl -i` / `tmux new-session` / `computer-use / Playwright` invocations, the paired CLEANUP block with cleanup receipt + `tmux kill-session`, and the leftover-state Stop rule so the mandates cannot be silently regressed. README and CHANGELOGs refreshed; stale 5,821-char claim replaced with measured 10,037 chars / 213 lines. All 9 ultrawork hook tests + 7 aggregate tests pass.
2026-05-27 13:07:20 +09:00
parent d18c2c078d
commit e746dcf0d8
7 changed files with 113 additions and 34 deletions
@@ -1,5 +1,13 @@
 # Changelog

+## Unreleased
+
+- Execution loop now mandates **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count.
+- New paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact (server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars) with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress.
+- New Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done.
+- Regression tests in `hooks/ultrawork-hooks.test.mjs` pin the SURFACE-AS-SCENARIO + CLEANUP mandates so they cannot be silently regressed.
+- Directive size: 10,037 chars across 213 lines.
+
 ## 0.1.0 — 2026-05-23

 Initial release.
@@ -7,11 +7,12 @@ Codex plugin that injects a compact orchestration directive (the **ultrawork** p
 | Mandate | Behavior |
 |---|---|
 | Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario's PASS condition is **observable evidence from the real surface** (`tmux` transcript, `curl` status+body, browser screenshot, Playwright assertion, computer-use action log, CLI stdout, parsed config dump, DB state diff). "Tests pass" alone is not evidence. |
+| Manual-QA-as-scenario + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) forces the agent to *actually invoke* the surface end-to-end with concrete commands keyed by surface (HTTP → `curl -i`, TUI → `tmux new-session` + `send-keys` + `capture-pane`, GUI → computer-use / Playwright, CLI, DB diff). Step 5 (**CLEANUP, PAIRED**) requires teardown of every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. |
 | Durable /tmp notepad | `mktemp -t ulw-$(date +%Y%m%d-%H%M%S).XXXXXX.md` with sections `Plan`, `Success criteria + QA scenarios`, `Now`, `Todo`, `Findings`, `Learnings`. **Append**, never rewrite. |
 | Obsessive atomic todos | Every action — even one-line edits, `ls`, single test runs — becomes a todo. Format: `path: <action> for <criterion> — verify by <check>`. One in_progress at a time, mark completed immediately. |
 | GPT-5.2 xhigh verification gate | Triggered automatically on user-requested rigor, 3+ files, 20+ turns, 30+ minutes, or refactor/migration/perf/security work. Use the bundled `codex-ultrawork-reviewer` agent role when available. Reviewer verdict is **binding** — no "false positive", no minimising, no arguing. Loop until **unconditional** approval. "Looks good but…" = REJECTION. |

-The directive is currently 5,821 chars (was 7,761) and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules).
+The directive is currently 10,037 chars / 213 lines and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules).

 ## Install (via this marketplace)

@@ -58,9 +58,12 @@ The criteria MUST list, upfront:
  1. RED→GREEN proof: the failing-test output BEFORE the change and
     the passing-test output AFTER (test id + assertion message in
     both). Tests added AFTER the green code do NOT satisfy this.
-  2. Real-surface artifact — `tmux` session transcript, `curl` status
-     + body, browser screenshot / Playwright assertion, computer-use
-     action log, CLI stdout, parsed config dump, DB state diff.
+  2. Manual-QA-as-scenario artifact — you (the agent) actually
+     INVOKE the real surface, not just claim it would work: HTTP via
+     `curl` (status + body), terminal / TUI via `tmux` (new session,
+     send-keys, `capture-pane -p` transcript), GUI via computer-use
+     or Playwright (action log + screenshot), CLI stdout, DB state
+     diff. "Should respond" / "looks correct" is NOT QA.
  Tests are the FLOOR (required, never sufficient); the surface
  artifact is the CEILING (also required). "tests pass" alone is NOT
  done.
@@ -114,7 +117,7 @@ GOOD pair (test-first, ordered):
 BAD: "Implement feature" / "Fix bug" / "Add tests later" / writing
 production code before its failing test → rewrite.

-# Execution loop (strict TDD — RED → GREEN → SURFACE)
+# Execution loop (strict TDD — RED → GREEN → SURFACE → CLEAN)
 Until every success-criteria scenario PASSES with BOTH evidence pieces:
 1. Pick next criterion → mark in_progress → update notepad `## Now`.
 2. RED: write the failing test FIRST. Run it. Capture the exact
@@ -124,14 +127,35 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces:
 3. GREEN: write the SMALLEST production change that flips RED→GREEN.
   Re-run the test. Capture GREEN output. If GREEN required more than
   ~20 lines, your test was too coarse — split it.
-4. SURFACE: exercise the real user-facing surface named by the
-   criterion (tmux / curl / browser / computer-use / CLI / DB).
-   Capture the artifact path into the notepad.
-5. Verify: LSP diagnostics clean on changed files + full test suite
+4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS):
+   ACTUALLY invoke the surface end-to-end as the user would. Concrete
+   moves keyed by surface (use the one the criterion names):
+     • HTTP → `curl -i` the running server; capture status + body.
+     • Terminal / TUI / interactive CLI → `tmux new-session -d -s
+       ulw-qa-<criterion>`, drive with `send-keys`, dump via
+       `tmux capture-pane -pS -E -`.
+     • GUI / web → computer-use / Playwright; capture action log +
+       screenshot path.
+     • Pure CLI → run it; capture stdout + exit code.
+     • DB / state-mutating → before/after diff.
+   Paste the artifact path into the notepad. `--dry-run`, printing the
+   command, or "looks correct" does NOT count.
+5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA
+   spawned in step 4 MUST be torn down before this step completes:
+   server PIDs (`kill <pid>`; verify `kill -0` fails), `tmux` sessions
+   (`tmux kill-session -t ulw-qa-<criterion>`; verify with `tmux ls`),
+   browser / Playwright contexts (`.close()`), containers
+   (`docker rm -f`), bound ports (`lsof -i :<port>` empty), temp
+   sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env
+   vars. Append a one-line cleanup receipt to the notepad next to the
+   artifact, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo;
+   rm -rf /tmp/ulw.aB12cD`. No receipt → criterion stays in_progress.
+6. Verify: LSP diagnostics clean on changed files + full test suite
   green (no skipped, no xfail added this turn).
-6. Mark completed. Append non-obvious findings / learnings.
-7. After each increment, re-run the FULL scenario list. Record
-   PASS/FAIL inline with BOTH evidence paths. Loop until all PASS.
+7. Mark completed. Append non-obvious findings / learnings.
+8. After each increment, re-run the FULL scenario list. Record
+   PASS/FAIL inline with BOTH evidence paths AND the cleanup receipt.
+   Loop until all PASS.

 Parallel-batch independent reads / searches / subagents within a step,
 but NEVER parallelise RED and GREEN of the same criterion.
@@ -198,8 +222,13 @@ message + present for approval.
  list (`<sha> <subject>`). No file-by-file changelog unless asked.

 # Stop rules
- Stop ONLY when every scenario PASSES with captured evidence, notepad
-  is current, and (if gate triggered) reviewer approved unconditionally.
+- Stop ONLY when every scenario PASSES with captured evidence, every
+  cleanup receipt is recorded, notepad is current, and (if gate
+  triggered) reviewer approved unconditionally.
+- Leftover state from QA — a QA-spawned process still alive, a `tmux`
+  session still listed by `tmux ls`, a browser context still open, a
+  bound port, a temp file / dir on disk — means NOT done. Tear it
+  down, record the receipt, then continue.
 - After 2 identical failed attempts at one step, surface what was tried
  and ask the user before another retry.
 - After 2 parallel exploration waves yield no new useful facts, stop
@@ -119,6 +119,40 @@ test("#given ultrawork prompt #when detector runs #then directive keeps goal bud
 	assert.doesNotMatch(result.stdout, /200000/i);
 });

+test("#given ultrawork prompt #when detector runs #then directive mandates manual-QA-as-scenario for http/tmux/computer-use", async () => {
+	const payload = JSON.stringify({
+		hook_event_name: "UserPromptSubmit",
+		prompt: "please ultrawork",
+	});
+
+	const result = await runPython(detectorPath, payload);
+
+	assert.equal(result.code, 0);
+	assert.equal(result.stderr, "");
+	assert.match(result.stdout, /SURFACE-AS-SCENARIO/);
+	assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/);
+	assert.match(result.stdout, /curl -i/);
+	assert.match(result.stdout, /tmux new-session/);
+	assert.match(result.stdout, /computer-use \/ Playwright/);
+});
+
+test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => {
+	const payload = JSON.stringify({
+		hook_event_name: "UserPromptSubmit",
+		prompt: "please ultrawork",
+	});
+
+	const result = await runPython(detectorPath, payload);
+
+	assert.equal(result.code, 0);
+	assert.equal(result.stderr, "");
+	assert.match(result.stdout, /CLEANUP \(PAIRED \u2014 NEVER SKIP\)/);
+	assert.match(result.stdout, /cleanup receipt/);
+	assert.match(result.stdout, /tmux kill-session/);
+	assert.match(result.stdout, /Leftover state from QA/);
+	assert.match(result.stdout, /means NOT done/);
+});
+
 test("#given identifier-like ulw #when detector runs #then does not emit directive", async () => {
 	const payload = JSON.stringify({
 		hook_event_name: "UserPromptSubmit",