feat(omo-codex): enumerate 4 manual-QA channels + forbid tests-alone verification in ultrawork/ultragoal
Promote the four real-usage QA channels (HTTP call, tmux, Browser use, Computer use) into a dedicated top-level `# Manual-QA channels` section in both the Codex ultrawork directive and the ultragoal skill. Each channel ships with the concrete invocation (`curl -i`, `tmux new-session` + `send-keys` + `capture-pane`, Playwright / puppeteer / Chromium, OS-level GUI automation) and the artifact to capture. The Goal section now declares `TESTS ALONE NEVER PROVE DONE` and requires every criterion to build a fresh real-usage scenario and run it through one of the four channels every time. Auxiliary surfaces (CLI stdout / DB diff / parsed config dump) are explicitly demoted: they only satisfy genuinely CLI- or data-shaped criteria, never user-facing behavior. Bootstrap criterion item 2 and execution step 4 used to repeat the same surface enumeration; they now collapse onto the new channel table (single source of truth, less drift). EXECUTE-AS-SCENARIO in the ultragoal skill follows the same compression. The previous CLEANUP (paired teardown + receipt) + leftover-state stop rule from the prior commit are preserved. Regression tests in `components/ultrawork/hooks/ultrawork-hooks.test.mjs` pin: the `# Manual-QA channels` heading, all four channel labels (`HTTP call`, `tmux`, `Browser use`, `Computer use`), `TESTS ALONE NEVER PROVE DONE`, `every criterion needs its own real-usage scenario`, and `every time`. README + CHANGELOGs document the restructure and the new size (11,005 chars / 232 lines). All 11 ultrawork hook tests + 7 aggregate tests pass. Installed locally via `runCodexInstaller` into `~/.codex/plugins/cache/code-yeongyu-codex-plugins/omo/0.1.0/`; cache hooks emit the new directive end-to-end and the identifier-like `ulw_helper.ts` keyword-boundary check still suppresses.
This commit is contained in:
YeonGyu-Kim
@@ -3,4 +3,5 @@
|
||||
## [0.1.0] - unreleased
|
||||
|
||||
- Initial scaffold of codex-ultragoal plugin.
|
||||
- Per-Criterion Cycle: `EXECUTE` is now **EXECUTE-AS-SCENARIO** — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). Inserted a new **CLEAN (PAIRED, NEVER SKIP)** step that tears down every QA-spawned process / `tmux` session / browser context / container / port / temp dir before recording evidence; the cleanup receipt is embedded in the `--evidence` string. Missing receipt → record BLOCKED, not PASS. Added Constraint #13 and a Stop Rule for leftover state.
|
||||
- Per-Criterion Cycle: `EXECUTE` is now **EXECUTE-AS-SCENARIO** — the agent must run the Manual-QA channel scenario the criterion named (HTTP call / tmux / browser use / computer use; see new `## Manual-QA channels` section). Inserted a new **CLEAN (PAIRED, NEVER SKIP)** step that tears down every QA-spawned process / `tmux` session / browser context / container / port / temp dir before recording evidence; the cleanup receipt is embedded in the `--evidence` string. Missing receipt → record BLOCKED, not PASS. Added Constraint #13 and a Stop Rule for leftover state.
|
||||
- New top-level **`## Manual-QA channels`** section explicitly enumerates the four channels (HTTP call, tmux, Browser use, Computer use) with concrete commands and required artifacts. Goal section now declares **TESTS ALONE NEVER PROVE DONE**: a green test suite is supporting evidence, never completion proof. Criterion-refinement step 2 requires each criterion to name its channel up front.
|
||||
|
||||
@@ -9,9 +9,20 @@ Use GPT-5.x style: outcome-first, evidence-bound, atomic decisions, no nested br
|
||||
|
||||
## Goal
|
||||
Deliver every goal in `.omo/ultragoal/goals.json` end-to-end.
|
||||
Prove EVERY success criterion with captured observable evidence from the real surface.
|
||||
Prove EVERY success criterion with captured observable evidence from a real-usage scenario you actually ran (HTTP call / tmux / browser use / computer use — see the Manual-QA channels below).
|
||||
TESTS ALONE NEVER PROVE DONE. A green test suite is supporting evidence, not completion proof.
|
||||
Audit each pass, fail, block, steering change, and checkpoint in `.omo/ultragoal/ledger.jsonl`.
|
||||
|
||||
## Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT)
|
||||
For every criterion, build a real-usage scenario through ONE of these four channels and run it yourself before recording PASS. The full test suite being green is NEVER verification on its own.
|
||||
|
||||
1. **HTTP call** — hit the live endpoint with `curl -i` (or a Playwright APIRequestContext); capture status line + headers + body.
|
||||
2. **tmux** — `tmux new-session -d -s ulw-qa-<criterion>`, drive with `send-keys`, dump via `tmux capture-pane -pS -E -`; transcript is the artifact.
|
||||
3. **Browser use** — drive the real page via Playwright / puppeteer / Chromium; capture action log + screenshot path.
|
||||
4. **Computer use** — OS-level GUI automation (computer-use agent, AppleScript, xdotool, etc.) against the running app; capture action log + screenshot.
|
||||
|
||||
Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config dump) satisfy CLI- or data-shaped criteria but NEVER replace a channel scenario for user-facing behavior. `--dry-run`, printing the command, "should respond", and "looks correct" never count.
|
||||
|
||||
## Artifacts
|
||||
- `.omo/ultragoal/brief.md`: original brief and durable constraints.
|
||||
- `.omo/ultragoal/goals.json`: goals with embedded `successCriteria` per goal.
|
||||
@@ -34,12 +45,12 @@ Write state through the CLI path. Do not hand-edit state files.
|
||||
### 2. Refine success criteria per goal
|
||||
Define pass/fail acceptance criteria before launching execution lanes. Include the command, artifact, or manual check that will prove success.
|
||||
Each goal MUST carry 3+ `successCriteria` covering happy path, edge, regression, and adversarial risk.
|
||||
For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, and stop condition.
|
||||
For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, stop condition, and the Manual-QA channel (HTTP call / tmux / browser use / computer use) that will exercise it.
|
||||
Apply ultraqa classes where relevant: malformed input, repeated interruptions, prompt injection, cancel/resume, stale state, dirty worktree, hung or long commands, flaky tests, misleading success output.
|
||||
Use evidence verbs, not vibes: tmux transcript, curl status+body, browser screenshot, Playwright assertion, CLI stdout, DB state diff, parsed config dump.
|
||||
"Tests pass" is supporting signal, not completion proof.
|
||||
Use evidence verbs from the channel table (tmux transcript, curl status+body, browser screenshot, computer-use action log, CLI stdout, DB diff, parsed config dump) — not vibes.
|
||||
"Tests pass" is supporting signal, NEVER completion proof. Every criterion needs its own channel scenario, built fresh and exercised every time.
|
||||
Record manual QA notes when behavior is user-visible.
|
||||
Revise any criterion that lacks observable `expectedEvidence` before execution.
|
||||
Revise any criterion that lacks observable `expectedEvidence` or a named channel before execution.
|
||||
|
||||
### 3. Inspect state
|
||||
Run `omo ultragoal status --json`.
|
||||
@@ -64,7 +75,7 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a
|
||||
### Per-Criterion Cycle
|
||||
1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds.
|
||||
2. Register atomic todos: `path: <action> for <criterion> - verify by <check>`.
|
||||
3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-<criterion>` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count.
|
||||
3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY run the Manual-QA channel scenario the criterion named (HTTP call / tmux / browser use / computer use — see the channel table above). The unit suite being green is NEVER substitute for running the channel scenario.
|
||||
4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump.
|
||||
5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-<criterion>`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :<port>` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS.
|
||||
6. RECORD exactly one result:
|
||||
|
||||
@@ -2,11 +2,18 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Execution loop now mandates **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count.
|
||||
- New paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact (server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars) with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress.
|
||||
- New Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done.
|
||||
- Regression tests in `hooks/ultrawork-hooks.test.mjs` pin the SURFACE-AS-SCENARIO + CLEANUP mandates so they cannot be silently regressed.
|
||||
- Directive size: 10,037 chars across 213 lines.
|
||||
- New top-level **`# Manual-QA channels`** section explicitly enumerates the four real-usage channels the agent MUST verify through: (1) HTTP call, (2) tmux, (3) Browser use, (4) Computer use — each with concrete commands and the artifact to capture. Auxiliary surfaces (CLI stdout / DB diff / parsed config dump) only count for genuinely CLI- or data-shaped criteria.
|
||||
- Goal section now shouts **TESTS ALONE NEVER PROVE DONE**: a green test suite is supporting evidence, never completion proof. Every criterion needs its own real-usage scenario, built fresh and run through one of the four channels, every time.
|
||||
- Bootstrap criterion item 2 and execution step 4 collapse onto the new channel table to remove triple-enumeration of the same surfaces (single source of truth, less drift).
|
||||
- Execution loop step 4 (**SURFACE-AS-SCENARIO**) runs the chosen channel scenario; step 5 (**CLEANUP, PAIRED**) tears down server PIDs, `tmux` sessions, browser / Playwright contexts, containers, bound ports, temp files / dirs, QA-only env vars and records a one-line receipt. Missing receipt → criterion stays in_progress. Leftover state from QA = NOT done (Stop rule).
|
||||
- Regression tests in `hooks/ultrawork-hooks.test.mjs` now pin: the four channel labels (`HTTP call`, `tmux`, `Browser use`, `Computer use`), `TESTS ALONE NEVER PROVE DONE`, `every criterion needs its own real-usage scenario`, the `# Manual-QA channels` heading, plus SURFACE-AS-SCENARIO + CLEANUP + leftover-state stop rule.
|
||||
- Directive size: 11,005 chars across 232 lines.
|
||||
|
||||
### Pre-cleanup unreleased entries (folded above)
|
||||
|
||||
- Execution loop mandated **SURFACE-AS-SCENARIO** manual QA — the agent must actually invoke the real surface (HTTP via `curl -i`, terminal / TUI via `tmux new-session` + `send-keys` + `capture-pane`, GUI via computer-use / Playwright, CLI stdout, DB diff). `--dry-run` and "looks correct" no longer count.
|
||||
- Paired **CLEANUP** step requires teardown of every QA-spawned runtime artifact with a one-line cleanup receipt recorded in the notepad. Missing receipt → criterion stays in_progress.
|
||||
- Stop rule: leftover state from QA (live process, `tmux` session, browser context, bound port, temp dir) means NOT done.
|
||||
|
||||
## 0.1.0 — 2026-05-23
|
||||
|
||||
|
||||
@@ -6,13 +6,14 @@ Codex plugin that injects a compact orchestration directive (the **ultrawork** p
|
||||
|
||||
| Mandate | Behavior |
|
||||
|---|---|
|
||||
| Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario's PASS condition is **observable evidence from the real surface** (`tmux` transcript, `curl` status+body, browser screenshot, Playwright assertion, computer-use action log, CLI stdout, parsed config dump, DB state diff). "Tests pass" alone is not evidence. |
|
||||
| Manual-QA-as-scenario + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) forces the agent to *actually invoke* the surface end-to-end with concrete commands keyed by surface (HTTP → `curl -i`, TUI → `tmux new-session` + `send-keys` + `capture-pane`, GUI → computer-use / Playwright, CLI, DB diff). Step 5 (**CLEANUP, PAIRED**) requires teardown of every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. |
|
||||
| Goal + binding success criteria | Call `create_goal` (or open with a `# Goal` block) listing the deliverable + **3+ realistic QA scenarios** (happy path, edge cases, adjacent-surface regression). Each scenario MUST name which **Manual-QA channel** it will use. "Tests pass" is supporting signal, NEVER completion proof. |
|
||||
| Manual-QA channels (TESTS ALONE NEVER PROVE DONE) | A dedicated top-level section enumerates the **four** channels you can use to verify a criterion in reality: **(1) HTTP call** (`curl -i` / Playwright APIRequestContext), **(2) tmux** (`tmux new-session` + `send-keys` + `capture-pane`), **(3) Browser use** (Playwright / puppeteer / Chromium driving the real page), **(4) Computer use** (OS-level GUI automation against the running app). Every criterion picks one channel, builds a real-usage scenario, runs it, and captures the artifact — every time. Aux surfaces (CLI stdout / DB diff / parsed config) only count for genuinely CLI- or data-shaped criteria. |
|
||||
| Surface + paired cleanup | Execution loop step 4 (**SURFACE-AS-SCENARIO**) runs the chosen channel scenario end-to-end. Step 5 (**CLEANUP, PAIRED**) tears down every QA-spawned process / tmux session / browser context / container / port / temp dir, with a one-line receipt appended to the notepad. Leftover state → NOT done. |
|
||||
| Durable /tmp notepad | `mktemp -t ulw-$(date +%Y%m%d-%H%M%S).XXXXXX.md` with sections `Plan`, `Success criteria + QA scenarios`, `Now`, `Todo`, `Findings`, `Learnings`. **Append**, never rewrite. |
|
||||
| Obsessive atomic todos | Every action — even one-line edits, `ls`, single test runs — becomes a todo. Format: `path: <action> for <criterion> — verify by <check>`. One in_progress at a time, mark completed immediately. |
|
||||
| GPT-5.2 xhigh verification gate | Triggered automatically on user-requested rigor, 3+ files, 20+ turns, 30+ minutes, or refactor/migration/perf/security work. Use the bundled `codex-ultrawork-reviewer` agent role when available. Reviewer verdict is **binding** — no "false positive", no minimising, no arguing. Loop until **unconditional** approval. "Looks good but…" = REJECTION. |
|
||||
|
||||
The directive is currently 10,037 chars / 213 lines and follows the GPT-5.5 prompting structure (Role / Goal / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules).
|
||||
The directive is currently 11,005 chars / 232 lines and follows the GPT-5.5 prompting structure (Role / Goal / Manual-QA channels / Bootstrap / Execution loop / Verification gate / Commits / Constraints / Output / Stop rules).
|
||||
|
||||
## Install (via this marketplace)
|
||||
|
||||
|
||||
@@ -35,9 +35,37 @@ narration.
|
||||
|
||||
# Goal
|
||||
Deliver EXACTLY what the user asked, end-to-end working, proven by
|
||||
(a) a test written test-first that went RED→GREEN and (b) manual QA
|
||||
from the real surface with captured observable evidence. BOTH gates,
|
||||
every change, no exceptions.
|
||||
(a) a test written test-first that went RED→GREEN and (b) a manual-QA
|
||||
scenario you actually run against the real surface (HTTP call / tmux /
|
||||
browser use / computer use — see the channel table below) with the
|
||||
artifact captured. Both gates, every change, no exceptions.
|
||||
TESTS ALONE NEVER PROVE DONE. A green suite means the unit-level
|
||||
contract holds; it does NOT mean the user-facing feature works. Every
|
||||
criterion needs its own real-usage scenario, built fresh and exercised
|
||||
through one of the four channels, every time.
|
||||
|
||||
# Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT)
|
||||
For every criterion, build a real-usage scenario through ONE of these
|
||||
four channels and run it yourself before declaring the criterion done.
|
||||
The full test suite being green is NEVER verification on its own.
|
||||
|
||||
1. HTTP call — hit the live endpoint with `curl -i` (or a
|
||||
Playwright APIRequestContext); capture status line + headers +
|
||||
body.
|
||||
2. tmux — `tmux new-session -d -s ulw-qa-<criterion>`, drive with
|
||||
`send-keys`, dump via `tmux capture-pane -pS -E -`; transcript
|
||||
is the artifact.
|
||||
3. Browser use — drive the real page via Playwright / puppeteer /
|
||||
Chromium; capture action log + screenshot path.
|
||||
4. Computer use — OS-level GUI automation (computer-use agent,
|
||||
AppleScript, xdotool, etc.) against the running app; capture
|
||||
action log + screenshot.
|
||||
|
||||
Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config
|
||||
dump) are valid evidence when the criterion is genuinely CLI- or
|
||||
data-shaped, but they do NOT replace a channel scenario for any
|
||||
user-facing behavior. `--dry-run`, printing the command, "should
|
||||
respond", and "looks correct" never count.
|
||||
|
||||
# Bootstrap (DO ALL THREE BEFORE ANY OTHER WORK — NO SKIPPING)
|
||||
|
||||
@@ -58,15 +86,13 @@ The criteria MUST list, upfront:
|
||||
1. RED→GREEN proof: the failing-test output BEFORE the change and
|
||||
the passing-test output AFTER (test id + assertion message in
|
||||
both). Tests added AFTER the green code do NOT satisfy this.
|
||||
2. Manual-QA-as-scenario artifact — you (the agent) actually
|
||||
INVOKE the real surface, not just claim it would work: HTTP via
|
||||
`curl` (status + body), terminal / TUI via `tmux` (new session,
|
||||
send-keys, `capture-pane -p` transcript), GUI via computer-use
|
||||
or Playwright (action log + screenshot), CLI stdout, DB state
|
||||
diff. "Should respond" / "looks correct" is NOT QA.
|
||||
Tests are the FLOOR (required, never sufficient); the surface
|
||||
artifact is the CEILING (also required). "tests pass" alone is NOT
|
||||
done.
|
||||
2. Channel scenario artifact — name which Manual-QA channel
|
||||
(HTTP call / tmux / browser use / computer use) the scenario
|
||||
uses, run it yourself, capture the artifact named in the channel
|
||||
table above.
|
||||
Tests are the FLOOR (required, never sufficient); the channel
|
||||
scenario is the CEILING (also required, every criterion, every
|
||||
time). "tests pass" alone is NEVER done.
|
||||
|
||||
These scenarios are the contract. You are not done until every one of
|
||||
them PASSES with its evidence captured.
|
||||
@@ -128,18 +154,11 @@ Until every success-criteria scenario PASSES with BOTH evidence pieces:
|
||||
Re-run the test. Capture GREEN output. If GREEN required more than
|
||||
~20 lines, your test was too coarse — split it.
|
||||
4. SURFACE-AS-SCENARIO (MANUAL QA — YOU EXECUTE IT, NO STUBS):
|
||||
ACTUALLY invoke the surface end-to-end as the user would. Concrete
|
||||
moves keyed by surface (use the one the criterion names):
|
||||
• HTTP → `curl -i` the running server; capture status + body.
|
||||
• Terminal / TUI / interactive CLI → `tmux new-session -d -s
|
||||
ulw-qa-<criterion>`, drive with `send-keys`, dump via
|
||||
`tmux capture-pane -pS -E -`.
|
||||
• GUI / web → computer-use / Playwright; capture action log +
|
||||
screenshot path.
|
||||
• Pure CLI → run it; capture stdout + exit code.
|
||||
• DB / state-mutating → before/after diff.
|
||||
Paste the artifact path into the notepad. `--dry-run`, printing the
|
||||
command, or "looks correct" does NOT count.
|
||||
Run the Manual-QA channel scenario the criterion named (HTTP
|
||||
call / tmux / browser use / computer use; see the channel table at
|
||||
the top). Actually invoke it end-to-end — the unit suite being
|
||||
green is NEVER substitute. Paste the artifact path into the
|
||||
notepad.
|
||||
5. CLEANUP (PAIRED — NEVER SKIP): every runtime artifact the QA
|
||||
spawned in step 4 MUST be torn down before this step completes:
|
||||
server PIDs (`kill <pid>`; verify `kill -0` fails), `tmux` sessions
|
||||
|
||||
@@ -133,7 +133,39 @@ test("#given ultrawork prompt #when detector runs #then directive mandates manua
|
||||
assert.match(result.stdout, /MANUAL QA \u2014 YOU EXECUTE IT, NO STUBS/);
|
||||
assert.match(result.stdout, /curl -i/);
|
||||
assert.match(result.stdout, /tmux new-session/);
|
||||
assert.match(result.stdout, /computer-use \/ Playwright/);
|
||||
});
|
||||
|
||||
test("#given ultrawork prompt #when detector runs #then directive enumerates 4 manual-QA channels explicitly", async () => {
|
||||
const payload = JSON.stringify({
|
||||
hook_event_name: "UserPromptSubmit",
|
||||
prompt: "please ultrawork",
|
||||
});
|
||||
|
||||
const result = await runPython(detectorPath, payload);
|
||||
|
||||
assert.equal(result.code, 0);
|
||||
assert.equal(result.stderr, "");
|
||||
assert.match(result.stdout, /# Manual-QA channels/);
|
||||
assert.match(result.stdout, /PICK ONE PER CRITERION \u2014 ACTUALLY RUN IT/);
|
||||
assert.match(result.stdout, /1\. HTTP call/);
|
||||
assert.match(result.stdout, /2\. tmux/);
|
||||
assert.match(result.stdout, /3\. Browser use/);
|
||||
assert.match(result.stdout, /4\. Computer use/);
|
||||
});
|
||||
|
||||
test("#given ultrawork prompt #when detector runs #then directive forbids tests-alone verification", async () => {
|
||||
const payload = JSON.stringify({
|
||||
hook_event_name: "UserPromptSubmit",
|
||||
prompt: "please ultrawork",
|
||||
});
|
||||
|
||||
const result = await runPython(detectorPath, payload);
|
||||
|
||||
assert.equal(result.code, 0);
|
||||
assert.equal(result.stderr, "");
|
||||
assert.match(result.stdout, /TESTS ALONE NEVER PROVE DONE/);
|
||||
assert.match(result.stdout, /Every[\s\n]+criterion needs its own real-usage scenario/);
|
||||
assert.match(result.stdout, /every time/);
|
||||
});
|
||||
|
||||
test("#given ultrawork prompt #when detector runs #then directive mandates paired cleanup with receipt and leftover-state stop rule", async () => {
|
||||
|
||||
@@ -9,9 +9,20 @@ Use GPT-5.x style: outcome-first, evidence-bound, atomic decisions, no nested br
|
||||
|
||||
## Goal
|
||||
Deliver every goal in `.omo/ultragoal/goals.json` end-to-end.
|
||||
Prove EVERY success criterion with captured observable evidence from the real surface.
|
||||
Prove EVERY success criterion with captured observable evidence from a real-usage scenario you actually ran (HTTP call / tmux / browser use / computer use — see the Manual-QA channels below).
|
||||
TESTS ALONE NEVER PROVE DONE. A green test suite is supporting evidence, not completion proof.
|
||||
Audit each pass, fail, block, steering change, and checkpoint in `.omo/ultragoal/ledger.jsonl`.
|
||||
|
||||
## Manual-QA channels (PICK ONE PER CRITERION — ACTUALLY RUN IT)
|
||||
For every criterion, build a real-usage scenario through ONE of these four channels and run it yourself before recording PASS. The full test suite being green is NEVER verification on its own.
|
||||
|
||||
1. **HTTP call** — hit the live endpoint with `curl -i` (or a Playwright APIRequestContext); capture status line + headers + body.
|
||||
2. **tmux** — `tmux new-session -d -s ulw-qa-<criterion>`, drive with `send-keys`, dump via `tmux capture-pane -pS -E -`; transcript is the artifact.
|
||||
3. **Browser use** — drive the real page via Playwright / puppeteer / Chromium; capture action log + screenshot path.
|
||||
4. **Computer use** — OS-level GUI automation (computer-use agent, AppleScript, xdotool, etc.) against the running app; capture action log + screenshot.
|
||||
|
||||
Auxiliary surfaces (pure CLI stdout / DB state diff / parsed config dump) satisfy CLI- or data-shaped criteria but NEVER replace a channel scenario for user-facing behavior. `--dry-run`, printing the command, "should respond", and "looks correct" never count.
|
||||
|
||||
## Artifacts
|
||||
- `.omo/ultragoal/brief.md`: original brief and durable constraints.
|
||||
- `.omo/ultragoal/goals.json`: goals with embedded `successCriteria` per goal.
|
||||
@@ -34,12 +45,12 @@ Write state through the CLI path. Do not hand-edit state files.
|
||||
### 2. Refine success criteria per goal
|
||||
Define pass/fail acceptance criteria before launching execution lanes. Include the command, artifact, or manual check that will prove success.
|
||||
Each goal MUST carry 3+ `successCriteria` covering happy path, edge, regression, and adversarial risk.
|
||||
For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, and stop condition.
|
||||
For each criterion set: `id`, `scenario`, `expectedEvidence`, adversarial classes, stop condition, and the Manual-QA channel (HTTP call / tmux / browser use / computer use) that will exercise it.
|
||||
Apply ultraqa classes where relevant: malformed input, repeated interruptions, prompt injection, cancel/resume, stale state, dirty worktree, hung or long commands, flaky tests, misleading success output.
|
||||
Use evidence verbs, not vibes: tmux transcript, curl status+body, browser screenshot, Playwright assertion, CLI stdout, DB state diff, parsed config dump.
|
||||
"Tests pass" is supporting signal, not completion proof.
|
||||
Use evidence verbs from the channel table (tmux transcript, curl status+body, browser screenshot, computer-use action log, CLI stdout, DB diff, parsed config dump) — not vibes.
|
||||
"Tests pass" is supporting signal, NEVER completion proof. Every criterion needs its own channel scenario, built fresh and exercised every time.
|
||||
Record manual QA notes when behavior is user-visible.
|
||||
Revise any criterion that lacks observable `expectedEvidence` before execution.
|
||||
Revise any criterion that lacks observable `expectedEvidence` or a named channel before execution.
|
||||
|
||||
### 3. Inspect state
|
||||
Run `omo ultragoal status --json`.
|
||||
@@ -64,7 +75,7 @@ Loop per goal. Cap at 5 cycles per goal. Cap identical same-criterion failures a
|
||||
### Per-Criterion Cycle
|
||||
1. PLAN: read `criterion.scenario`, `criterion.expectedEvidence`, prior ledger entries, and safety bounds.
|
||||
2. Register atomic todos: `path: <action> for <criterion> - verify by <check>`.
|
||||
3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY invoke the real surface end-to-end as the user would. Concrete moves: HTTP via `curl -i` (status + body); terminal / TUI via a dedicated `tmux new-session -d -s ulw-qa-<criterion>` driven with `send-keys` and dumped via `tmux capture-pane -pS -E -`; GUI via computer-use / Playwright (action log + screenshot); pure CLI via running it (stdout + exit); DB via before/after diff. `--dry-run`, printing the command, or "should respond" does NOT count.
|
||||
3. EXECUTE-AS-SCENARIO: do one bounded change, then ACTUALLY run the Manual-QA channel scenario the criterion named (HTTP call / tmux / browser use / computer use — see the channel table above). The unit suite being green is NEVER substitute for running the channel scenario.
|
||||
4. CAPTURE: collect the observable artifact path: transcript, stdout, screenshot, assertion, status+body, diff, or parsed dump.
|
||||
5. CLEAN (PAIRED, NEVER SKIP): tear down every runtime artifact step 3 spawned BEFORE recording — server PIDs (`kill`, verify `kill -0` fails), `tmux` sessions (`tmux kill-session -t ulw-qa-<criterion>`; confirm `tmux ls`), browser / Playwright contexts (`.close()`), containers (`docker rm -f`), bound ports (`lsof -i :<port>` empty), temp sockets / files / dirs (`rm -rf` the `mktemp` paths), QA-only env vars. Embed a one-line cleanup receipt in the evidence string, e.g. `cleanup: killed 12345; tmux kill-session ulw-qa-foo; rm -rf /tmp/ulw.aB12cD`. Missing receipt → record BLOCKED, not PASS.
|
||||
6. RECORD exactly one result:
|
||||
|
||||
Reference in New Issue
Block a user