feat(omo-codex): enumerate 4 manual-QA channels + forbid tests-alone verification in ultrawork/ultragoal

Promote the four real-usage QA channels (HTTP call, tmux, Browser use, Computer use) into a dedicated top-level `# Manual-QA channels` section in both the Codex ultrawork directive and the ultragoal skill. Each channel ships with the concrete invocation (`curl -i`, `tmux new-session` + `send-keys` + `capture-pane`, Playwright / puppeteer / Chromium, OS-level GUI automation) and the artifact to capture. The Goal section now declares `TESTS ALONE NEVER PROVE DONE` and requires every criterion to build a fresh real-usage scenario and run it through one of the four channels every time. Auxiliary surfaces (CLI stdout / DB diff / parsed config dump) are explicitly demoted: they only satisfy genuinely CLI- or data-shaped criteria, never user-facing behavior.

Bootstrap criterion item 2 and execution step 4 used to repeat the same surface enumeration; they now collapse onto the new channel table (single source of truth, less drift). EXECUTE-AS-SCENARIO in the ultragoal skill follows the same compression. The previous CLEANUP (paired teardown + receipt) + leftover-state stop rule from the prior commit are preserved.

Regression tests in `components/ultrawork/hooks/ultrawork-hooks.test.mjs` pin: the `# Manual-QA channels` heading, all four channel labels (`HTTP call`, `tmux`, `Browser use`, `Computer use`), `TESTS ALONE NEVER PROVE DONE`, `every criterion needs its own real-usage scenario`, and `every time`. README + CHANGELOGs document the restructure and the new size (11,005 chars / 232 lines). All 11 ultrawork hook tests + 7 aggregate tests pass. Installed locally via `runCodexInstaller` into `~/.codex/plugins/cache/code-yeongyu-codex-plugins/omo/0.1.0/`; cache hooks emit the new directive end-to-end and the identifier-like `ulw_helper.ts` keyword-boundary check still suppresses.
This commit is contained in:
YeonGyu-Kim
2026-05-27 13:26:10 +09:00
parent e746dcf0d8
commit 73e8988f31
7 changed files with 128 additions and 46 deletions
@@ -35,9 +35,37 @@ narration.
# Goal
Deliver EXACTLY what the user asked, end-to-end working, proven by
(a) a test written test-first that went RED→GREEN and (b) manual QA
from the real surface with captured observable evidence. BOTH gates,
every change, no exceptions.
(a) a test written test-first that went RED→GREEN and (b) a manual-QA
scenario you actually run against the real surface (HTTP call / tmux /
browser use / computer use — see the channel table below) with the
artifact captured. Both gates, every change, no exceptions.
TESTS ALONE NEVER PROVE DONE. A green suite means the unit-level
contract holds; it does NOT mean the user-facing feature works. Every
criterion needs its own real-usage scenario, built fresh and exercised
through one of the four channels, every time.
# Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT)
For every criterion, build a real-usage scenario through ONE of these
four channels and run it yourself before declaring the criterion done.
The full test suite being green is NEVER verification on its own.
1. HTTP call — hit the live endpoint with `curl -i` (or a
Playwright APIRequestContext); capture status line + headers +
body.
2. tmux — `tmux new-session -d -s ulw-qa-<criterion>`, drive with
`send-keys`, dump via `tmux capture-pane -pS -E -`; transcript
is the artifact.
3. Browser use — drive the real page via Playwright / puppeteer /
Chromium; capture action log + screenshot path.
4. Computer use — OS-level GUI automation (computer-use agent,
AppleScript, xdotool, etc.) against the running app; capture
action log + screenshot.
Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config
dump) are valid evidence when the criterion is genuinely CLI- or
data-shaped, but they do NOT replace a channel scenario for any
user-facing behavior. `--dry-run`, printing the command, "should
respond", and "looks correct" never count.
# Bootstrap (DO ALL THREE BEFORE ANY OTHER WORK — NO SKIPPING)
@@ -58,15 +86,13 @@ The criteria MUST list, upfront:
1. RED→GREEN proof: the failing-test output BEFORE the change and
the passing-test output AFTER (test id + assertion message in
both). Tests added AFTER the green code do NOT satisfy this.
2. Manual-QA-as-scenario artifact — you (the agent) actually
INVOKE the real surface, not just claim it would work: HTTP via
`curl` (status + body), terminal / TUI via `tmux` (new session,
send-keys, `capture-pane -p` transcript), GUI via computer-use
or Playwright (action log + screenshot), CLI stdout, DB state
diff. "Should respond" / "looks correct" is NOT QA.
Tests are the FLOOR (required, never sufficient); the surface
artifact is the CEILING (also required). "tests pass" alone is NOT
done.
2. Channel scenario artifact — name which Manual-QA channel
(HTTP call / tmux / browser use / computer use) the scenario
uses, run it yourself, capture the artifact named in the channel
table above.
Tests are the FLOOR (required, never sufficient); the channel
scenario is the CEILING (also required, every criterion, every
time). "tests pass" alone is NEVER done.
These scenarios are the contract. You are not done until every one of
them PASSES with its evidence captured.
@@ -128,18 +154,11 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces:
Re-run the test. Capture GREEN output. If GREEN required more than
~20 lines, your test was too coarse — split it.
4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS):
ACTUALLY invoke the surface end-to-end as the user would. Concrete
moves keyed by surface (use the one the criterion names):
• HTTP → `curl -i` the running server; capture status + body.
• Terminal / TUI / interactive CLI → `tmux new-session -d -s
ulw-qa-<criterion>`, drive with `send-keys`, dump via
`tmux capture-pane -pS -E -`.
• GUI / web → computer-use / Playwright; capture action log +
screenshot path.
• Pure CLI → run it; capture stdout + exit code.
• DB / state-mutating → before/after diff.
Paste the artifact path into the notepad. `--dry-run`, printing the
command, or "looks correct" does NOT count.
Run the Manual-QA channel scenario the criterion named (HTTP
call / tmux / browser use / computer use; see the channel table at
the top). Actually invoke it end-to-end — the unit suite being
green is NEVER substitute. Paste the artifact path into the
notepad.
5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA
spawned in step 4 MUST be torn down before this step completes:
server PIDs (`kill <pid>`; verify `kill -0` fails), `tmux` sessions
@@ -133,7 +133,39 @@ test("#given ultrawork prompt #when detector runs #then directive mandates manua
assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/);
assert.match(result.stdout, /curl -i/);
assert.match(result.stdout, /tmux new-session/);
assert.match(result.stdout, /computer-use \/ Playwright/);
});
test("#given ultrawork prompt #when detector runs #then directive enumerates 4 manual-QA channels explicitly", async () => {
const payload = JSON.stringify({
hook_event_name: "UserPromptSubmit",
prompt: "please ultrawork",
});
const result = await runPython(detectorPath, payload);
assert.equal(result.code, 0);
assert.equal(result.stderr, "");
assert.match(result.stdout, /# Manual-QA channels/);
assert.match(result.stdout, /PICK ONE PER CRITERION \u2014 ACTUALLY RUN IT/);
assert.match(result.stdout, /1\. HTTP call/);
assert.match(result.stdout, /2\. tmux/);
assert.match(result.stdout, /3\. Browser use/);
assert.match(result.stdout, /4\. Computer use/);
});
test("#given ultrawork prompt #when detector runs #then directive forbids tests-alone verification", async () => {
const payload = JSON.stringify({
hook_event_name: "UserPromptSubmit",
prompt: "please ultrawork",
});
const result = await runPython(detectorPath, payload);
assert.equal(result.code, 0);
assert.equal(result.stderr, "");
assert.match(result.stdout, /TESTS ALONE NEVER PROVE DONE/);
assert.match(result.stdout, /Every[\s\n]+criterion needs its own real-usage scenario/);
assert.match(result.stdout, /every time/);
});
test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => {