feat(omo-codex): mandate manual-QA-as-scenario + paired cleanup in ultrawork/ultragoal

Both the Codex ultrawork directive and the ultragoal skill now force the agent to actually invoke the real user-facing surface (HTTP via `curl -i`, terminal/TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff) instead of treating evidence as a free-form artifact list. A paired CLEANUP step requires teardown of every QA-spawned process, tmux session, browser context, container, bound port, temp file/dir, and QA-only env var, with a one-line cleanup receipt recorded next to the artifact path (ultrawork) or embedded in the `--evidence` string (ultragoal). Missing receipt keeps the criterion in_progress / records BLOCKED. New Stop rule: leftover state from QA means NOT done.

Regression tests in `components/ultrawork/hooks/ultrawork-hooks.test.mjs` pin SURFACE-AS-SCENARIO, the concrete `curl -i` / `tmux new-session` / `computer-use / Playwright` invocations, the paired CLEANUP block with cleanup receipt + `tmux kill-session`, and the leftover-state Stop rule so the mandates cannot be silently regressed. README and CHANGELOGs refreshed; stale 5,821-char claim replaced with measured 10,037 chars / 213 lines. All 9 ultrawork hook tests + 7 aggregate tests pass.

packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md

@@ -3,3 +3,4 @@
 ## [0.1.0] - unreleased
 
 - Initial scaffold of codex-ultragoal plugin.
+- Per-Criterion Cycle: `EXECUTE` is now **EXECUTE-AS-SCENARIO** — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). Inserted a new **CLEAN (PAIRED, NEVER SKIP)** step that tears down every QA-spawned process / `tmux` session / browser context / container / port / temp dir before recording evidence; the cleanup receipt is embedded in the `--evidence` string. Missing receipt → record BLOCKED, not PASS. Added Constraint #13 and a Stop Rule for leftover state.

packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md

@@ -64,16 +64,17 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a
 ### Per-Criterion Cycle
 1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds.
 2. Register atomic todos: `path: <action> for <criterion> - verify by <check>`.
-3. EXECUTE: do one bounded change or check, then exercise the real surface named by the criterion.
+3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-<criterion>` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count.
-4. CAPTURE: collect actual observable evidence: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump.
+4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump.
-5. RECORD exactly one result:
+5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-<criterion>`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :<port>` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS.
-   - PASS: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status pass --evidence "<observable>" --json`
+6. RECORD exactly one result:
-   - FAIL: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status fail --evidence "<observable>" --notes "<diagnosis>" --json`
+   - PASS: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status pass --evidence "<observable> | <cleanup receipt>" --json`
-   - BLOCKED: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status blocked --evidence "<observable>" --notes "<safety/blocker>" --json`
+   - FAIL: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status fail --evidence "<observable> | <cleanup receipt>" --notes "<diagnosis>" --json`
-6. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion.
+   - BLOCKED: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status blocked --evidence "<observable>" --notes "<safety/blocker/leftover-state>" --json`
-7. After 3 same-criterion failures, exit the goal with diagnosis.
+7. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion (including a fresh cleanup).
-8. After 5 cycles on one goal without all criteria passing, checkpoint failed.
+8. After 3 same-criterion failures, exit the goal with diagnosis.
-9. Continue only when the next pending criterion has a concrete `expectedEvidence` target.
+9. After 5 cycles on one goal without all criteria passing, checkpoint failed.
+10. Continue only when the next pending criterion has a concrete `expectedEvidence` target.
 
 ### Goal Completion
 1. Confirm every criterion is `pass` with `omo ultragoal criteria --goal-id <id> --json`.
@@ -133,6 +134,7 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr
 10. Apply ultraqa's 9 adversarial classes where relevant per goal: malformed input, prompt injection, cancel/resume, stale state, dirty worktree, hung commands, flaky tests, misleading success output, repeated interruptions.
 11. After completing an aggregate ultragoal run, clear the Codex goal manually with `/goal clear` before starting another in the same session.
 12. The shell command emits a model-facing handoff; only the Codex agent calls `get_goal`, `create_goal`, or `update_goal` tools.
+13. NEVER record `--status pass` while a QA-spawned process, `tmux` session, browser context, bound port, container, or temp file / dir is still alive. The evidence string MUST include the cleanup receipt. Leftover runtime state = BLOCKED, not PASS.
 
 ## Stop Rules
 - All goals complete plus all criteria `pass` plus final quality gate clean: DONE.
@@ -140,4 +142,5 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr
 - 5 cycles on one goal without all-pass: checkpoint failed, surface.
 - Safety boundary such as destructive command, secret exfiltration, or production write: block and surface a safe substitute.
 - Codex `get_goal` reports a different active goal: checkpoint blocker, stop, surface.
+- Leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir): NOT pass. Clean up, append the receipt, then continue.
 - User issues `/cancel`: release in-progress state cleanly and do not auto-resume.

packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## Unreleased
+
+- Execution loop now mandates **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count.
+- New paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact (server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars) with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress.
+- New Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done.
+- Regression tests in `hooks/ultrawork-hooks.test.mjs` pin the SURFACE-AS-SCENARIO + CLEANUP mandates so they cannot be silently regressed.
+- Directive size: 10,037 chars across 213 lines.
+
 ## 0.1.0 — 2026-05-23
 
 Initial release.

packages/omo-codex/plugin/components/ultrawork/README.md

@@ -7,11 +7,12 @@ Codex plugin that injects a compact orchestration directive (the **ultrawork** p
 | Mandate | Behavior |
 |---|---|
 | Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario's PASS condition is **observable evidence from the real surface** (`tmux` transcript, `curl` status+body, browser screenshot, Playwright assertion, computer-use action log, CLI stdout, parsed config dump, DB state diff). "Tests pass" alone is not evidence. |
+| Manual-QA-as-scenario + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) forces the agent to *actually invoke* the surface end-to-end with concrete commands keyed by surface (HTTP → `curl -i`, TUI → `tmux new-session` + `send-keys` + `capture-pane`, GUI → computer-use / Playwright, CLI, DB diff). Step 5 (**CLEANUP, PAIRED**) requires teardown of every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. |
 | Durable /tmp notepad | `mktemp -t ulw-$(date +%Y%m%d-%H%M%S).XXXXXX.md` with sections `Plan`, `Success criteria + QA scenarios`, `Now`, `Todo`, `Findings`, `Learnings`. **Append**, never rewrite. |
 | Obsessive atomic todos | Every action — even one-line edits, `ls`, single test runs — becomes a todo. Format: `path: <action> for <criterion> — verify by <check>`. One in_progress at a time, mark completed immediately. |
 | GPT-5.2 xhigh verification gate | Triggered automatically on user-requested rigor, 3+ files, 20+ turns, 30+ minutes, or refactor/migration/perf/security work. Use the bundled `codex-ultrawork-reviewer` agent role when available. Reviewer verdict is **binding** — no "false positive", no minimising, no arguing. Loop until **unconditional** approval. "Looks good but…" = REJECTION. |
 
-The directive is currently 5,821 chars (was 7,761) and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules).
+The directive is currently 10,037 chars / 213 lines and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules).
 
 ## Install (via this marketplace)
 

packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py

@@ -58,9 +58,12 @@ The criteria MUST list, upfront:
   1. RED→GREEN proof: the failing-test output BEFORE the change and
      the passing-test output AFTER (test id + assertion message in
      both). Tests added AFTER the green code do NOT satisfy this.
-  2. Real-surface artifact — `tmux` session transcript, `curl` status
+  2. Manual-QA-as-scenario artifact — you (the agent) actually
-     + body, browser screenshot / Playwright assertion, computer-use
+     INVOKE the real surface, not just claim it would work: HTTP via
-     action log, CLI stdout, parsed config dump, DB state diff.
+     `curl` (status + body), terminal / TUI via `tmux` (new session,
+     send-keys, `capture-pane -p` transcript), GUI via computer-use
+     or Playwright (action log + screenshot), CLI stdout, DB state
+     diff. "Should respond" / "looks correct" is NOT QA.
   Tests are the FLOOR (required, never sufficient); the surface
   artifact is the CEILING (also required). "tests pass" alone is NOT
   done.
@@ -114,7 +117,7 @@ GOOD pair (test-first, ordered):
 BAD: "Implement feature" / "Fix bug" / "Add tests later" / writing
 production code before its failing test → rewrite.
 
-# Execution loop (strict TDD — RED → GREEN → SURFACE)
+# Execution loop (strict TDD — RED → GREEN → SURFACE → CLEAN)
 Until every success-criteria scenario PASSES with BOTH evidence pieces:
 1. Pick next criterion → mark in_progress → update notepad `## Now`.
 2. RED: write the failing test FIRST. Run it. Capture the exact
@@ -124,14 +127,35 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces:
 3. GREEN: write the SMALLEST production change that flips RED→GREEN.
    Re-run the test. Capture GREEN output. If GREEN required more than
    ~20 lines, your test was too coarse — split it.
-4. SURFACE: exercise the real user-facing surface named by the
+4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS):
-   criterion (tmux / curl / browser / computer-use / CLI / DB).
+   ACTUALLY invoke the surface end-to-end as the user would. Concrete
-   Capture the artifact path into the notepad.
+   moves keyed by surface (use the one the criterion names):
-5. Verify: LSP diagnostics clean on changed files + full test suite
+     • HTTP → `curl -i` the running server; capture status + body.
+     • Terminal / TUI / interactive CLI → `tmux new-session -d -s
+       ulw-qa-<criterion>`, drive with `send-keys`, dump via
+       `tmux capture-pane -pS -E -`.
+     • GUI / web → computer-use / Playwright; capture action log +
+       screenshot path.
+     • Pure CLI → run it; capture stdout + exit code.
+     • DB / state-mutating → before/after diff.
+   Paste the artifact path into the notepad. `--dry-run`, printing the
+   command, or "looks correct" does NOT count.
+5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA
+   spawned in step 4 MUST be torn down before this step completes:
+   server PIDs (`kill <pid>`; verify `kill -0` fails), `tmux` sessions
+   (`tmux kill-session -t ulw-qa-<criterion>`; verify with `tmux ls`),
+   browser / Playwright contexts (`.close()`), containers
+   (`docker rm -f`), bound ports (`lsof -i :<port>` empty), temp
+   sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env
+   vars. Append a one-line cleanup receipt to the notepad next to the
+   artifact, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo;
+   rm -rf /tmp/ulw.aB12cD`. No receipt → criterion stays in_progress.
+6. Verify: LSP diagnostics clean on changed files + full test suite
    green (no skipped, no xfail added this turn).
-6. Mark completed. Append non-obvious findings / learnings.
+7. Mark completed. Append non-obvious findings / learnings.
-7. After each increment, re-run the FULL scenario list. Record
+8. After each increment, re-run the FULL scenario list. Record
-   PASS/FAIL inline with BOTH evidence paths. Loop until all PASS.
+   PASS/FAIL inline with BOTH evidence paths AND the cleanup receipt.
+   Loop until all PASS.
 
 Parallel-batch independent reads / searches / subagents within a step,
 but NEVER parallelise RED and GREEN of the same criterion.
@@ -198,8 +222,13 @@ message + present for approval.
   list (`<sha> <subject>`). No file-by-file changelog unless asked.
 
 # Stop rules
-- Stop ONLY when every scenario PASSES with captured evidence, notepad
+- Stop ONLY when every scenario PASSES with captured evidence, every
-  is current, and (if gate triggered) reviewer approved unconditionally.
+  cleanup receipt is recorded, notepad is current, and (if gate
+  triggered) reviewer approved unconditionally.
+- Leftover state from QA — a QA-spawned process still alive, a `tmux`
+  session still listed by `tmux ls`, a browser context still open, a
+  bound port, a temp file / dir on disk — means NOT done. Tear it
+  down, record the receipt, then continue.
 - After 2 identical failed attempts at one step, surface what was tried
   and ask the user before another retry.
 - After 2 parallel exploration waves yield no new useful facts, stop

packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs

@@ -119,6 +119,40 @@ test("#given ultrawork prompt #when detector runs #then directive keeps goal bud
 	assert.doesNotMatch(result.stdout, /200000/i);
 });
 
+test("#given ultrawork prompt #when detector runs #then directive mandates manual-QA-as-scenario for http/tmux/computer-use", async () => {
+	const payload = JSON.stringify({
+		hook_event_name: "UserPromptSubmit",
+		prompt: "please ultrawork",
+	});
+
+	const result = await runPython(detectorPath, payload);
+
+	assert.equal(result.code, 0);
+	assert.equal(result.stderr, "");
+	assert.match(result.stdout, /SURFACE-AS-SCENARIO/);
+	assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/);
+	assert.match(result.stdout, /curl -i/);
+	assert.match(result.stdout, /tmux new-session/);
+	assert.match(result.stdout, /computer-use \/ Playwright/);
+});
+
+test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => {
+	const payload = JSON.stringify({
+		hook_event_name: "UserPromptSubmit",
+		prompt: "please ultrawork",
+	});
+
+	const result = await runPython(detectorPath, payload);
+
+	assert.equal(result.code, 0);
+	assert.equal(result.stderr, "");
+	assert.match(result.stdout, /CLEANUP \(PAIRED \u2014 NEVER SKIP\)/);
+	assert.match(result.stdout, /cleanup receipt/);
+	assert.match(result.stdout, /tmux kill-session/);
+	assert.match(result.stdout, /Leftover state from QA/);
+	assert.match(result.stdout, /means NOT done/);
+});
+
 test("#given identifier-like ulw #when detector runs #then does not emit directive", async () => {
 	const payload = JSON.stringify({
 		hook_event_name: "UserPromptSubmit",

packages/omo-codex/plugin/skills/ultragoal/SKILL.md

@@ -64,16 +64,17 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a
 ### Per-Criterion Cycle
 1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds.
 2. Register atomic todos: `path: <action> for <criterion> - verify by <check>`.
-3. EXECUTE: do one bounded change or check, then exercise the real surface named by the criterion.
+3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-<criterion>` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count.
-4. CAPTURE: collect actual observable evidence: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump.
+4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump.
-5. RECORD exactly one result:
+5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-<criterion>`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :<port>` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS.
-   - PASS: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status pass --evidence "<observable>" --json`
+6. RECORD exactly one result:
-   - FAIL: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status fail --evidence "<observable>" --notes "<diagnosis>" --json`
+   - PASS: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status pass --evidence "<observable> | <cleanup receipt>" --json`
-   - BLOCKED: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status blocked --evidence "<observable>" --notes "<safety/blocker>" --json`
+   - FAIL: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status fail --evidence "<observable> | <cleanup receipt>" --notes "<diagnosis>" --json`
-6. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion.
+   - BLOCKED: `omo ultragoal record-evidence --goal-id <id> --criterion-id <id> --status blocked --evidence "<observable>" --notes "<safety/blocker/leftover-state>" --json`
-7. After 3 same-criterion failures, exit the goal with diagnosis.
+7. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion (including a fresh cleanup).
-8. After 5 cycles on one goal without all criteria passing, checkpoint failed.
+8. After 3 same-criterion failures, exit the goal with diagnosis.
-9. Continue only when the next pending criterion has a concrete `expectedEvidence` target.
+9. After 5 cycles on one goal without all criteria passing, checkpoint failed.
+10. Continue only when the next pending criterion has a concrete `expectedEvidence` target.
 
 ### Goal Completion
 1. Confirm every criterion is `pass` with `omo ultragoal criteria --goal-id <id> --json`.
@@ -133,6 +134,7 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr
 10. Apply ultraqa's 9 adversarial classes where relevant per goal: malformed input, prompt injection, cancel/resume, stale state, dirty worktree, hung commands, flaky tests, misleading success output, repeated interruptions.
 11. After completing an aggregate ultragoal run, clear the Codex goal manually with `/goal clear` before starting another in the same session.
 12. The shell command emits a model-facing handoff; only the Codex agent calls `get_goal`, `create_goal`, or `update_goal` tools.
+13. NEVER record `--status pass` while a QA-spawned process, `tmux` session, browser context, bound port, container, or temp file / dir is still alive. The evidence string MUST include the cleanup receipt. Leftover runtime state = BLOCKED, not PASS.
 
 ## Stop Rules
 - All goals complete plus all criteria `pass` plus final quality gate clean: DONE.
@@ -140,4 +142,5 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr
 - 5 cycles on one goal without all-pass: checkpoint failed, surface.
 - Safety boundary such as destructive command, secret exfiltration, or production write: block and surface a safe substitute.
 - Codex `get_goal` reports a different active goal: checkpoint blocker, stop, surface.
+- Leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir): NOT pass. Clean up, append the receipt, then continue.
 - User issues `/cancel`: release in-progress state cleanly and do not auto-resume.