diff --git a/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md b/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md index 0d9de92ad..688781ba0 100644 --- a/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md +++ b/packages/omo-codex/plugin/components/ultragoal/CHANGELOG.md @@ -3,3 +3,4 @@ ## [0.1.0] - unreleased - Initial scaffold of codex-ultragoal plugin. +- Per-Criterion Cycle: `EXECUTE` is now **EXECUTE-AS-SCENARIO** — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). Inserted a new **CLEAN (PAIRED, NEVER SKIP)** step that tears down every QA-spawned process / `tmux` session / browser context / container / port / temp dir before recording evidence; the cleanup receipt is embedded in the `--evidence` string. Missing receipt → record BLOCKED, not PASS. Added Constraint #13 and a Stop Rule for leftover state. diff --git a/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md b/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md index b047917d4..eb8d70e03 100644 --- a/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md +++ b/packages/omo-codex/plugin/components/ultragoal/skills/ultragoal/SKILL.md @@ -64,16 +64,17 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a ### Per-Criterion Cycle 1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds. 2. Register atomic todos: `path: for - verify by `. -3. EXECUTE: do one bounded change or check, then exercise the real surface named by the criterion. -4. CAPTURE: collect actual observable evidence: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump. -5. RECORD exactly one result: - - PASS: `omo ultragoal record-evidence --goal-id --criterion-id --status pass --evidence "" --json` - - FAIL: `omo ultragoal record-evidence --goal-id --criterion-id --status fail --evidence "" --notes "" --json` - - BLOCKED: `omo ultragoal record-evidence --goal-id --criterion-id --status blocked --evidence "" --notes "" --json` -6. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion. -7. After 3 same-criterion failures, exit the goal with diagnosis. -8. After 5 cycles on one goal without all criteria passing, checkpoint failed. -9. Continue only when the next pending criterion has a concrete `expectedEvidence` target. +3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count. +4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump. +5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS. +6. RECORD exactly one result: + - PASS: `omo ultragoal record-evidence --goal-id --criterion-id --status pass --evidence " | " --json` + - FAIL: `omo ultragoal record-evidence --goal-id --criterion-id --status fail --evidence " | " --notes "" --json` + - BLOCKED: `omo ultragoal record-evidence --goal-id --criterion-id --status blocked --evidence "" --notes "" --json` +7. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion (including a fresh cleanup). +8. After 3 same-criterion failures, exit the goal with diagnosis. +9. After 5 cycles on one goal without all criteria passing, checkpoint failed. +10. Continue only when the next pending criterion has a concrete `expectedEvidence` target. ### Goal Completion 1. Confirm every criterion is `pass` with `omo ultragoal criteria --goal-id --json`. @@ -133,6 +134,7 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr 10. Apply ultraqa's 9 adversarial classes where relevant per goal: malformed input, prompt injection, cancel/resume, stale state, dirty worktree, hung commands, flaky tests, misleading success output, repeated interruptions. 11. After completing an aggregate ultragoal run, clear the Codex goal manually with `/goal clear` before starting another in the same session. 12. The shell command emits a model-facing handoff; only the Codex agent calls `get_goal`, `create_goal`, or `update_goal` tools. +13. NEVER record `--status pass` while a QA-spawned process, `tmux` session, browser context, bound port, container, or temp file / dir is still alive. The evidence string MUST include the cleanup receipt. Leftover runtime state = BLOCKED, not PASS. ## Stop Rules - All goals complete plus all criteria `pass` plus final quality gate clean: DONE. @@ -140,4 +142,5 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr - 5 cycles on one goal without all-pass: checkpoint failed, surface. - Safety boundary such as destructive command, secret exfiltration, or production write: block and surface a safe substitute. - Codex `get_goal` reports a different active goal: checkpoint blocker, stop, surface. +- Leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir): NOT pass. Clean up, append the receipt, then continue. - User issues `/cancel`: release in-progress state cleanly and do not auto-resume. diff --git a/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md b/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md index 8244d1a40..6776a832a 100644 --- a/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md +++ b/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## Unreleased + +- Execution loop now mandates **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count. +- New paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact (server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars) with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress. +- New Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done. +- Regression tests in `hooks/ultrawork-hooks.test.mjs` pin the SURFACE-AS-SCENARIO + CLEANUP mandates so they cannot be silently regressed. +- Directive size: 10,037 chars across 213 lines. + ## 0.1.0 — 2026-05-23 Initial release. diff --git a/packages/omo-codex/plugin/components/ultrawork/README.md b/packages/omo-codex/plugin/components/ultrawork/README.md index 5f52d05e5..76dc8a0f7 100644 --- a/packages/omo-codex/plugin/components/ultrawork/README.md +++ b/packages/omo-codex/plugin/components/ultrawork/README.md @@ -7,11 +7,12 @@ Codex plugin that injects a compact orchestration directive (the **ultrawork** p | Mandate | Behavior | |---|---| | Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario's PASS condition is **observable evidence from the real surface** (`tmux` transcript, `curl` status+body, browser screenshot, Playwright assertion, computer-use action log, CLI stdout, parsed config dump, DB state diff). "Tests pass" alone is not evidence. | +| Manual-QA-as-scenario + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) forces the agent to *actually invoke* the surface end-to-end with concrete commands keyed by surface (HTTP → `curl -i`, TUI → `tmux new-session` + `send-keys` + `capture-pane`, GUI → computer-use / Playwright, CLI, DB diff). Step 5 (**CLEANUP, PAIRED**) requires teardown of every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. | | Durable /tmp notepad | `mktemp -t ulw-$(date +%Y%m%d-%H%M%S).XXXXXX.md` with sections `Plan`, `Success criteria + QA scenarios`, `Now`, `Todo`, `Findings`, `Learnings`. **Append**, never rewrite. | | Obsessive atomic todos | Every action — even one-line edits, `ls`, single test runs — becomes a todo. Format: `path: for — verify by `. One in_progress at a time, mark completed immediately. | | GPT-5.2 xhigh verification gate | Triggered automatically on user-requested rigor, 3+ files, 20+ turns, 30+ minutes, or refactor/migration/perf/security work. Use the bundled `codex-ultrawork-reviewer` agent role when available. Reviewer verdict is **binding** — no "false positive", no minimising, no arguing. Loop until **unconditional** approval. "Looks good but…" = REJECTION. | -The directive is currently 5,821 chars (was 7,761) and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules). +The directive is currently 10,037 chars / 213 lines and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules). ## Install (via this marketplace) diff --git a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py index 1452ff029..ee63838ba 100755 --- a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py +++ b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-detector.py @@ -58,9 +58,12 @@ The criteria MUST list, upfront: 1. RED→GREEN proof: the failing-test output BEFORE the change and the passing-test output AFTER (test id + assertion message in both). Tests added AFTER the green code do NOT satisfy this. - 2. Real-surface artifact — `tmux` session transcript, `curl` status - + body, browser screenshot / Playwright assertion, computer-use - action log, CLI stdout, parsed config dump, DB state diff. + 2. Manual-QA-as-scenario artifact — you (the agent) actually + INVOKE the real surface, not just claim it would work: HTTP via + `curl` (status + body), terminal / TUI via `tmux` (new session, + send-keys, `capture-pane -p` transcript), GUI via computer-use + or Playwright (action log + screenshot), CLI stdout, DB state + diff. "Should respond" / "looks correct" is NOT QA. Tests are the FLOOR (required, never sufficient); the surface artifact is the CEILING (also required). "tests pass" alone is NOT done. @@ -114,7 +117,7 @@ GOOD pair (test-first, ordered): BAD: "Implement feature" / "Fix bug" / "Add tests later" / writing production code before its failing test → rewrite. -# Execution loop (strict TDD — RED → GREEN → SURFACE) +# Execution loop (strict TDD — RED → GREEN → SURFACE → CLEAN) Until every success-criteria scenario PASSES with BOTH evidence pieces: 1. Pick next criterion → mark in_progress → update notepad `## Now`. 2. RED: write the failing test FIRST. Run it. Capture the exact @@ -124,14 +127,35 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces: 3. GREEN: write the SMALLEST production change that flips RED→GREEN. Re-run the test. Capture GREEN output. If GREEN required more than ~20 lines, your test was too coarse — split it. -4. SURFACE: exercise the real user-facing surface named by the - criterion (tmux / curl / browser / computer-use / CLI / DB). - Capture the artifact path into the notepad. -5. Verify: LSP diagnostics clean on changed files + full test suite +4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS): + ACTUALLY invoke the surface end-to-end as the user would. Concrete + moves keyed by surface (use the one the criterion names): + • HTTP → `curl -i` the running server; capture status + body. + • Terminal / TUI / interactive CLI → `tmux new-session -d -s + ulw-qa-`, drive with `send-keys`, dump via + `tmux capture-pane -pS -E -`. + • GUI / web → computer-use / Playwright; capture action log + + screenshot path. + • Pure CLI → run it; capture stdout + exit code. + • DB / state-mutating → before/after diff. + Paste the artifact path into the notepad. `--dry-run`, printing the + command, or "looks correct" does NOT count. +5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA + spawned in step 4 MUST be torn down before this step completes: + server PIDs (`kill `; verify `kill -0` fails), `tmux` sessions + (`tmux kill-session -t ulw-qa-`; verify with `tmux ls`), + browser / Playwright contexts (`.close()`), containers + (`docker rm -f`), bound ports (`lsof -i :` empty), temp + sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env + vars. Append a one-line cleanup receipt to the notepad next to the + artifact, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; + rm -rf /tmp/ulw.aB12cD`. No receipt → criterion stays in_progress. +6. Verify: LSP diagnostics clean on changed files + full test suite green (no skipped, no xfail added this turn). -6. Mark completed. Append non-obvious findings / learnings. -7. After each increment, re-run the FULL scenario list. Record - PASS/FAIL inline with BOTH evidence paths. Loop until all PASS. +7. Mark completed. Append non-obvious findings / learnings. +8. After each increment, re-run the FULL scenario list. Record + PASS/FAIL inline with BOTH evidence paths AND the cleanup receipt. + Loop until all PASS. Parallel-batch independent reads / searches / subagents within a step, but NEVER parallelise RED and GREEN of the same criterion. @@ -198,8 +222,13 @@ message + present for approval. list (` `). No file-by-file changelog unless asked. # Stop rules -- Stop ONLY when every scenario PASSES with captured evidence, notepad - is current, and (if gate triggered) reviewer approved unconditionally. +- Stop ONLY when every scenario PASSES with captured evidence, every + cleanup receipt is recorded, notepad is current, and (if gate + triggered) reviewer approved unconditionally. +- Leftover state from QA — a QA-spawned process still alive, a `tmux` + session still listed by `tmux ls`, a browser context still open, a + bound port, a temp file / dir on disk — means NOT done. Tear it + down, record the receipt, then continue. - After 2 identical failed attempts at one step, surface what was tried and ask the user before another retry. - After 2 parallel exploration waves yield no new useful facts, stop diff --git a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs index 6c8fba543..3f39e520a 100644 --- a/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs +++ b/packages/omo-codex/plugin/components/ultrawork/hooks/ultrawork-hooks.test.mjs @@ -119,6 +119,40 @@ test("#given ultrawork prompt #when detector runs #then directive keeps goal bud assert.doesNotMatch(result.stdout, /200000/i); }); +test("#given ultrawork prompt #when detector runs #then directive mandates manual-QA-as-scenario for http/tmux/computer-use", async () => { + const payload = JSON.stringify({ + hook_event_name: "UserPromptSubmit", + prompt: "please ultrawork", + }); + + const result = await runPython(detectorPath, payload); + + assert.equal(result.code, 0); + assert.equal(result.stderr, ""); + assert.match(result.stdout, /SURFACE-AS-SCENARIO/); + assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/); + assert.match(result.stdout, /curl -i/); + assert.match(result.stdout, /tmux new-session/); + assert.match(result.stdout, /computer-use \/ Playwright/); +}); + +test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => { + const payload = JSON.stringify({ + hook_event_name: "UserPromptSubmit", + prompt: "please ultrawork", + }); + + const result = await runPython(detectorPath, payload); + + assert.equal(result.code, 0); + assert.equal(result.stderr, ""); + assert.match(result.stdout, /CLEANUP \(PAIRED \u2014 NEVER SKIP\)/); + assert.match(result.stdout, /cleanup receipt/); + assert.match(result.stdout, /tmux kill-session/); + assert.match(result.stdout, /Leftover state from QA/); + assert.match(result.stdout, /means NOT done/); +}); + test("#given identifier-like ulw #when detector runs #then does not emit directive", async () => { const payload = JSON.stringify({ hook_event_name: "UserPromptSubmit", diff --git a/packages/omo-codex/plugin/skills/ultragoal/SKILL.md b/packages/omo-codex/plugin/skills/ultragoal/SKILL.md index b047917d4..eb8d70e03 100644 --- a/packages/omo-codex/plugin/skills/ultragoal/SKILL.md +++ b/packages/omo-codex/plugin/skills/ultragoal/SKILL.md @@ -64,16 +64,17 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a ### Per-Criterion Cycle 1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds. 2. Register atomic todos: `path: for - verify by `. -3. EXECUTE: do one bounded change or check, then exercise the real surface named by the criterion. -4. CAPTURE: collect actual observable evidence: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump. -5. RECORD exactly one result: - - PASS: `omo ultragoal record-evidence --goal-id --criterion-id --status pass --evidence "" --json` - - FAIL: `omo ultragoal record-evidence --goal-id --criterion-id --status fail --evidence "" --notes "" --json` - - BLOCKED: `omo ultragoal record-evidence --goal-id --criterion-id --status blocked --evidence "" --notes "" --json` -6. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion. -7. After 3 same-criterion failures, exit the goal with diagnosis. -8. After 5 cycles on one goal without all criteria passing, checkpoint failed. -9. Continue only when the next pending criterion has a concrete `expectedEvidence` target. +3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count. +4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump. +5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS. +6. RECORD exactly one result: + - PASS: `omo ultragoal record-evidence --goal-id --criterion-id --status pass --evidence " | " --json` + - FAIL: `omo ultragoal record-evidence --goal-id --criterion-id --status fail --evidence " | " --notes "" --json` + - BLOCKED: `omo ultragoal record-evidence --goal-id --criterion-id --status blocked --evidence "" --notes "" --json` +7. If actual does not match expected, diagnose, fix minimally, and rerun the SAME criterion (including a fresh cleanup). +8. After 3 same-criterion failures, exit the goal with diagnosis. +9. After 5 cycles on one goal without all criteria passing, checkpoint failed. +10. Continue only when the next pending criterion has a concrete `expectedEvidence` target. ### Goal Completion 1. Confirm every criterion is `pass` with `omo ultragoal criteria --goal-id --json`. @@ -133,6 +134,7 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr 10. Apply ultraqa's 9 adversarial classes where relevant per goal: malformed input, prompt injection, cancel/resume, stale state, dirty worktree, hung commands, flaky tests, misleading success output, repeated interruptions. 11. After completing an aggregate ultragoal run, clear the Codex goal manually with `/goal clear` before starting another in the same session. 12. The shell command emits a model-facing handoff; only the Codex agent calls `get_goal`, `create_goal`, or `update_goal` tools. +13. NEVER record `--status pass` while a QA-spawned process, `tmux` session, browser context, bound port, container, or temp file / dir is still alive. The evidence string MUST include the cleanup receipt. Leftover runtime state = BLOCKED, not PASS. ## Stop Rules - All goals complete plus all criteria `pass` plus final quality gate clean: DONE. @@ -140,4 +142,5 @@ Structured prompt directives accepted: `OMO_ULTRAGOAL_STEER: { ... }`, `omo.ultr - 5 cycles on one goal without all-pass: checkpoint failed, surface. - Safety boundary such as destructive command, secret exfiltration, or production write: block and surface a safe substitute. - Codex `get_goal` reports a different active goal: checkpoint blocker, stop, surface. +- Leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir): NOT pass. Clean up, append the receipt, then continue. - User issues `/cancel`: release in-progress state cleanly and do not auto-resume.