feat(omo-codex): enumerate 4 manual-QA channels + forbid tests-alone verification in ultrawork/ultragoal
Promote the four real-usage QA channels (HTTP call, tmux, Browser use, Computer use) into a dedicated top-level `# Manual-QA channels` section in both the Codex ultrawork directive and the ultragoal skill. Each channel ships with the concrete invocation (`curl -i`, `tmux new-session` + `send-keys` + `capture-pane`, Playwright / puppeteer / Chromium, OS-level GUI automation) and the artifact to capture. The Goal section now declares `TESTS ALONE NEVER PROVE DONE` and requires every criterion to build a fresh real-usage scenario and run it through one of the four channels every time. Auxiliary surfaces (CLI stdout / DB diff / parsed config dump) are explicitly demoted: they only satisfy genuinely CLI- or data-shaped criteria, never user-facing behavior. Bootstrap criterion item 2 and execution step 4 used to repeat the same surface enumeration; they now collapse onto the new channel table (single source of truth, less drift). EXECUTE-AS-SCENARIO in the ultragoal skill follows the same compression. The previous CLEANUP (paired teardown + receipt) + leftover-state stop rule from the prior commit are preserved. Regression tests in `components/ultrawork/hooks/ultrawork-hooks.test.mjs` pin: the `# Manual-QA channels` heading, all four channel labels (`HTTP call`, `tmux`, `Browser use`, `Computer use`), `TESTS ALONE NEVER PROVE DONE`, `every criterion needs its own real-usage scenario`, and `every time`. README + CHANGELOGs document the restructure and the new size (11,005 chars / 232 lines). All 11 ultrawork hook tests + 7 aggregate tests pass. Installed locally via `runCodexInstaller` into `~/.codex/plugins/cache/code-yeongyu-codex-plugins/omo/0.1.0/`; cache hooks emit the new directive end-to-end and the identifier-like `ulw_helper.ts` keyword-boundary check still suppresses.
This commit is contained in:
@@ -35,9 +35,37 @@ narration.
|
||||
|
||||
# Goal
|
||||
Deliver EXACTLY what the user asked, end-to-end working, proven by
|
||||
(a) a test written test-first that went RED→GREEN and (b) manual QA
|
||||
from the real surface with captured observable evidence. BOTH gates,
|
||||
every change, no exceptions.
|
||||
(a) a test written test-first that went RED→GREEN and (b) a manual-QA
|
||||
scenario you actually run against the real surface (HTTP call / tmux /
|
||||
browser use / computer use — see the channel table below) with the
|
||||
artifact captured. Both gates, every change, no exceptions.
|
||||
TESTS ALONE NEVER PROVE DONE. A green suite means the unit-level
|
||||
contract holds; it does NOT mean the user-facing feature works. Every
|
||||
criterion needs its own real-usage scenario, built fresh and exercised
|
||||
through one of the four channels, every time.
|
||||
|
||||
# Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT)
|
||||
For every criterion, build a real-usage scenario through ONE of these
|
||||
four channels and run it yourself before declaring the criterion done.
|
||||
The full test suite being green is NEVER verification on its own.
|
||||
|
||||
1. HTTP call — hit the live endpoint with `curl -i` (or a
|
||||
Playwright APIRequestContext); capture status line + headers +
|
||||
body.
|
||||
2. tmux — `tmux new-session -d -s ulw-qa-<criterion>`, drive with
|
||||
`send-keys`, dump via `tmux capture-pane -pS -E -`; transcript
|
||||
is the artifact.
|
||||
3. Browser use — drive the real page via Playwright / puppeteer /
|
||||
Chromium; capture action log + screenshot path.
|
||||
4. Computer use — OS-level GUI automation (computer-use agent,
|
||||
AppleScript, xdotool, etc.) against the running app; capture
|
||||
action log + screenshot.
|
||||
|
||||
Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config
|
||||
dump) are valid evidence when the criterion is genuinely CLI- or
|
||||
data-shaped, but they do NOT replace a channel scenario for any
|
||||
user-facing behavior. `--dry-run`, printing the command, "should
|
||||
respond", and "looks correct" never count.
|
||||
|
||||
# Bootstrap (DO ALL THREE BEFORE ANY OTHER WORK — NO SKIPPING)
|
||||
|
||||
@@ -58,15 +86,13 @@ The criteria MUST list, upfront:
|
||||
1. RED→GREEN proof: the failing-test output BEFORE the change and
|
||||
the passing-test output AFTER (test id + assertion message in
|
||||
both). Tests added AFTER the green code do NOT satisfy this.
|
||||
2. Manual-QA-as-scenario artifact — you (the agent) actually
|
||||
INVOKE the real surface, not just claim it would work: HTTP via
|
||||
`curl` (status + body), terminal / TUI via `tmux` (new session,
|
||||
send-keys, `capture-pane -p` transcript), GUI via computer-use
|
||||
or Playwright (action log + screenshot), CLI stdout, DB state
|
||||
diff. "Should respond" / "looks correct" is NOT QA.
|
||||
Tests are the FLOOR (required, never sufficient); the surface
|
||||
artifact is the CEILING (also required). "tests pass" alone is NOT
|
||||
done.
|
||||
2. Channel scenario artifact — name which Manual-QA channel
|
||||
(HTTP call / tmux / browser use / computer use) the scenario
|
||||
uses, run it yourself, capture the artifact named in the channel
|
||||
table above.
|
||||
Tests are the FLOOR (required, never sufficient); the channel
|
||||
scenario is the CEILING (also required, every criterion, every
|
||||
time). "tests pass" alone is NEVER done.
|
||||
|
||||
These scenarios are the contract. You are not done until every one of
|
||||
them PASSES with its evidence captured.
|
||||
@@ -128,18 +154,11 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces:
|
||||
Re-run the test. Capture GREEN output. If GREEN required more than
|
||||
~20 lines, your test was too coarse — split it.
|
||||
4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS):
|
||||
ACTUALLY invoke the surface end-to-end as the user would. Concrete
|
||||
moves keyed by surface (use the one the criterion names):
|
||||
• HTTP → `curl -i` the running server; capture status + body.
|
||||
• Terminal / TUI / interactive CLI → `tmux new-session -d -s
|
||||
ulw-qa-<criterion>`, drive with `send-keys`, dump via
|
||||
`tmux capture-pane -pS -E -`.
|
||||
• GUI / web → computer-use / Playwright; capture action log +
|
||||
screenshot path.
|
||||
• Pure CLI → run it; capture stdout + exit code.
|
||||
• DB / state-mutating → before/after diff.
|
||||
Paste the artifact path into the notepad. `--dry-run`, printing the
|
||||
command, or "looks correct" does NOT count.
|
||||
Run the Manual-QA channel scenario the criterion named (HTTP
|
||||
call / tmux / browser use / computer use; see the channel table at
|
||||
the top). Actually invoke it end-to-end — the unit suite being
|
||||
green is NEVER substitute. Paste the artifact path into the
|
||||
notepad.
|
||||
5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA
|
||||
spawned in step 4 MUST be torn down before this step completes:
|
||||
server PIDs (`kill <pid>`; verify `kill -0` fails), `tmux` sessions
|
||||
|
||||
@@ -133,7 +133,39 @@ test("#given ultrawork prompt #when detector runs #then directive mandates manua
|
||||
assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/);
|
||||
assert.match(result.stdout, /curl -i/);
|
||||
assert.match(result.stdout, /tmux new-session/);
|
||||
assert.match(result.stdout, /computer-use \/ Playwright/);
|
||||
});
|
||||
|
||||
test("#given ultrawork prompt #when detector runs #then directive enumerates 4 manual-QA channels explicitly", async () => {
|
||||
const payload = JSON.stringify({
|
||||
hook_event_name: "UserPromptSubmit",
|
||||
prompt: "please ultrawork",
|
||||
});
|
||||
|
||||
const result = await runPython(detectorPath, payload);
|
||||
|
||||
assert.equal(result.code, 0);
|
||||
assert.equal(result.stderr, "");
|
||||
assert.match(result.stdout, /# Manual-QA channels/);
|
||||
assert.match(result.stdout, /PICK ONE PER CRITERION \u2014 ACTUALLY RUN IT/);
|
||||
assert.match(result.stdout, /1\. HTTP call/);
|
||||
assert.match(result.stdout, /2\. tmux/);
|
||||
assert.match(result.stdout, /3\. Browser use/);
|
||||
assert.match(result.stdout, /4\. Computer use/);
|
||||
});
|
||||
|
||||
test("#given ultrawork prompt #when detector runs #then directive forbids tests-alone verification", async () => {
|
||||
const payload = JSON.stringify({
|
||||
hook_event_name: "UserPromptSubmit",
|
||||
prompt: "please ultrawork",
|
||||
});
|
||||
|
||||
const result = await runPython(detectorPath, payload);
|
||||
|
||||
assert.equal(result.code, 0);
|
||||
assert.equal(result.stderr, "");
|
||||
assert.match(result.stdout, /TESTS ALONE NEVER PROVE DONE/);
|
||||
assert.match(result.stdout, /Every[\s\n]+criterion needs its own real-usage scenario/);
|
||||
assert.match(result.stdout, /every time/);
|
||||
});
|
||||
|
||||
test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => {
|
||||
|
||||
Reference in New Issue
Block a user