feat(ultrawork): enforce manual QA execution and acceptance criteria workflow
Add MANUAL_QA_MANDATE sections to all three ultrawork prompts (default, GPT, Gemini). Agents must now define acceptance criteria in TODO/Task items before implementation, then execute manual QA themselves after completing work. lsp_diagnostics alone is explicitly called out as insufficient since it only catches type errors, not functional bugs. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
This commit is contained in:
@@ -202,7 +202,7 @@ BEFORE writing ANY code, you MUST define:
|
|||||||
| **Observable** | What can be measured/seen | "Console shows 'success', no errors" |
|
| **Observable** | What can be measured/seen | "Console shows 'success', no errors" |
|
||||||
| **Pass/Fail** | Binary, no ambiguity | "Returns 200 OK" not "should work" |
|
| **Pass/Fail** | Binary, no ambiguity | "Returns 200 OK" not "should work" |
|
||||||
|
|
||||||
Write these criteria explicitly. Share with user if scope is non-trivial.
|
Write these criteria explicitly. **Record them in your TODO/Task items.** Each task MUST include a "QA: [how to verify]" field. These criteria are your CONTRACT — work toward them, verify against them.
|
||||||
|
|
||||||
### Test Plan Template (MANDATORY for non-trivial tasks)
|
### Test Plan Template (MANDATORY for non-trivial tasks)
|
||||||
|
|
||||||
@@ -228,6 +228,32 @@ Write these criteria explicitly. Share with user if scope is non-trivial.
|
|||||||
|
|
||||||
**WITHOUT evidence = NOT verified = NOT done.**
|
**WITHOUT evidence = NOT verified = NOT done.**
|
||||||
|
|
||||||
|
<MANUAL_QA_MANDATE>
|
||||||
|
### YOU MUST EXECUTE MANUAL QA YOURSELF. THIS IS NOT OPTIONAL.
|
||||||
|
|
||||||
|
**YOUR FAILURE MODE**: You finish coding, run lsp_diagnostics, and declare "done" without actually TESTING the feature. lsp_diagnostics catches type errors, NOT functional bugs. Your work is NOT verified until you MANUALLY test it.
|
||||||
|
|
||||||
|
**WHAT MANUAL QA MEANS — execute ALL that apply:**
|
||||||
|
|
||||||
|
| If your change... | YOU MUST... |
|
||||||
|
|---|---|
|
||||||
|
| Adds/modifies a CLI command | Run the command with Bash. Show the output. |
|
||||||
|
| Changes build output | Run the build. Verify the output files exist and are correct. |
|
||||||
|
| Modifies API behavior | Call the endpoint. Show the response. |
|
||||||
|
| Changes UI rendering | Describe what renders. Use a browser tool if available. |
|
||||||
|
| Adds a new tool/hook/feature | Test it end-to-end in a real scenario. |
|
||||||
|
| Modifies config handling | Load the config. Verify it parses correctly. |
|
||||||
|
|
||||||
|
**UNACCEPTABLE QA CLAIMS:**
|
||||||
|
- "This should work" — RUN IT.
|
||||||
|
- "The types check out" — Types don't catch logic bugs. RUN IT.
|
||||||
|
- "lsp_diagnostics is clean" — That's a TYPE check, not a FUNCTIONAL check. RUN IT.
|
||||||
|
- "Tests pass" — Tests cover known cases. Does the ACTUAL FEATURE work as the user expects? RUN IT.
|
||||||
|
|
||||||
|
**You have Bash, you have tools. There is ZERO excuse for not running manual QA.**
|
||||||
|
**Manual QA is the FINAL gate before reporting completion. Skip it and your work is INCOMPLETE.**
|
||||||
|
</MANUAL_QA_MANDATE>
|
||||||
|
|
||||||
### TDD Workflow (when test infrastructure exists)
|
### TDD Workflow (when test infrastructure exists)
|
||||||
|
|
||||||
1. **SPEC**: Define what "working" means (success criteria above)
|
1. **SPEC**: Define what "working" means (success criteria above)
|
||||||
|
|||||||
@@ -236,6 +236,33 @@ task(subagent_type="plan", load_skills=[], prompt="<gathered context + user requ
|
|||||||
If ANY answer is no → GO BACK AND DO IT. Do not claim completion.
|
If ANY answer is no → GO BACK AND DO IT. Do not claim completion.
|
||||||
</ANTI_OPTIMISM_CHECKPOINT>
|
</ANTI_OPTIMISM_CHECKPOINT>
|
||||||
|
|
||||||
|
<MANUAL_QA_MANDATE>
|
||||||
|
### YOU MUST EXECUTE MANUAL QA. THIS IS NOT OPTIONAL. DO NOT SKIP THIS.
|
||||||
|
|
||||||
|
**YOUR FAILURE MODE**: You run lsp_diagnostics, see zero errors, and declare victory. lsp_diagnostics catches TYPE errors. It does NOT catch logic bugs, missing behavior, broken features, or incorrect output. Your work is NOT verified until you MANUALLY TEST the actual feature.
|
||||||
|
|
||||||
|
**AFTER every implementation, you MUST:**
|
||||||
|
|
||||||
|
1. **Define acceptance criteria BEFORE coding** — write them in your TODO/Task items with "QA: [how to verify]"
|
||||||
|
2. **Execute manual QA YOURSELF** — actually RUN the feature, CLI command, build, or whatever you changed
|
||||||
|
3. **Report what you observed** — show actual output, not claims
|
||||||
|
|
||||||
|
| If your change... | YOU MUST... |
|
||||||
|
|---|---|
|
||||||
|
| Adds/modifies a CLI command | Run the command with Bash. Show the output. |
|
||||||
|
| Changes build output | Run the build. Verify output files exist and are correct. |
|
||||||
|
| Modifies API behavior | Call the endpoint. Show the response. |
|
||||||
|
| Adds a new tool/hook/feature | Test it end-to-end in a real scenario. |
|
||||||
|
| Modifies config handling | Load the config. Verify it parses correctly. |
|
||||||
|
|
||||||
|
**UNACCEPTABLE (WILL BE REJECTED):**
|
||||||
|
- "This should work" — DID YOU RUN IT? NO? THEN RUN IT.
|
||||||
|
- "lsp_diagnostics is clean" — That is a TYPE check, not a FUNCTIONAL check. RUN THE FEATURE.
|
||||||
|
- "Tests pass" — Tests cover known cases. Does the ACTUAL feature work? VERIFY IT MANUALLY.
|
||||||
|
|
||||||
|
**You have Bash, you have tools. There is ZERO excuse for skipping manual QA.**
|
||||||
|
</MANUAL_QA_MANDATE>
|
||||||
|
|
||||||
**WITHOUT evidence = NOT verified = NOT done.**
|
**WITHOUT evidence = NOT verified = NOT done.**
|
||||||
|
|
||||||
## ZERO TOLERANCE FAILURES
|
## ZERO TOLERANCE FAILURES
|
||||||
|
|||||||
@@ -118,6 +118,14 @@ deep_context = background_output(task_id=...)
|
|||||||
- \`lsp_diagnostics\` on modified files
|
- \`lsp_diagnostics\` on modified files
|
||||||
- Run tests if available
|
- Run tests if available
|
||||||
|
|
||||||
|
## ACCEPTANCE CRITERIA WORKFLOW
|
||||||
|
|
||||||
|
**BEFORE implementation**, define what "done" means in concrete, binary terms:
|
||||||
|
|
||||||
|
1. Write acceptance criteria as pass/fail conditions (not "should work" — specific observable outcomes)
|
||||||
|
2. Record them in your TODO/Task items with a "QA: [how to verify]" field
|
||||||
|
3. Work toward those criteria, not just "finishing code"
|
||||||
|
|
||||||
## QUALITY STANDARDS
|
## QUALITY STANDARDS
|
||||||
|
|
||||||
| Phase | Action | Required Evidence |
|
| Phase | Action | Required Evidence |
|
||||||
@@ -125,6 +133,25 @@ deep_context = background_output(task_id=...)
|
|||||||
| Build | Run build command | Exit code 0 |
|
| Build | Run build command | Exit code 0 |
|
||||||
| Test | Execute test suite | All tests pass |
|
| Test | Execute test suite | All tests pass |
|
||||||
| Lint | Run lsp_diagnostics | Zero new errors |
|
| Lint | Run lsp_diagnostics | Zero new errors |
|
||||||
|
| **Manual QA** | **Execute the feature yourself** | **Actual output shown** |
|
||||||
|
|
||||||
|
<MANUAL_QA_MANDATE>
|
||||||
|
### MANUAL QA IS MANDATORY. lsp_diagnostics IS NOT ENOUGH.
|
||||||
|
|
||||||
|
lsp_diagnostics catches type errors. It does NOT catch logic bugs, missing behavior, or broken features. After EVERY implementation, you MUST manually test the actual feature.
|
||||||
|
|
||||||
|
**Execute ALL that apply:**
|
||||||
|
|
||||||
|
| If your change... | YOU MUST... |
|
||||||
|
|---|---|
|
||||||
|
| Adds/modifies a CLI command | Run the command with Bash. Show the output. |
|
||||||
|
| Changes build output | Run the build. Verify output files. |
|
||||||
|
| Modifies API behavior | Call the endpoint. Show the response. |
|
||||||
|
| Adds a new tool/hook/feature | Test it end-to-end in a real scenario. |
|
||||||
|
| Modifies config handling | Load the config. Verify it parses correctly. |
|
||||||
|
|
||||||
|
**"This should work" is NOT evidence. RUN IT. Show what happened. That is evidence.**
|
||||||
|
</MANUAL_QA_MANDATE>
|
||||||
|
|
||||||
## COMPLETION CRITERIA
|
## COMPLETION CRITERIA
|
||||||
|
|
||||||
@@ -133,6 +160,7 @@ A task is complete when:
|
|||||||
2. lsp_diagnostics shows zero errors on modified files
|
2. lsp_diagnostics shows zero errors on modified files
|
||||||
3. Tests pass (or pre-existing failures documented)
|
3. Tests pass (or pre-existing failures documented)
|
||||||
4. Code matches existing codebase patterns
|
4. Code matches existing codebase patterns
|
||||||
|
5. **Manual QA executed — actual feature tested, output observed and reported**
|
||||||
|
|
||||||
**Deliver exactly what was asked. No more, no less.**
|
**Deliver exactly what was asked. No more, no less.**
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user