fix(default-mode,multimodal-looker,delegate-task): preserve user-expected behavior

default-mode (system-transform):
- e5463e2db introduced auto-activation of ultrawork+ralph-loop, and
  dc2e082ac then skipped the ultrawork system prompt whenever ralph_loop
  was also enabled. Net effect: the keyword-detector still showed
  'Default ultrawork mode enabled' to the user, but the first turn had
  none of the ultrawork behavior. Loop continuation kept the ultrawork
  prefix, so the contract was honored only on later iterations.
- Drop the skip so the initial turn matches what the toast advertises.
  New matrix test pins all four (ultrawork, ralph_loop) combinations.

multimodal-looker:
- Prompt claimed 'read' and 'call_omo_agent' were available, but the
  look_at invocation runtime explicitly disables both via READ_ENABLED
  and createAgentToolAllowlist([]). Small VL models trusted the prompt
  and looped on rejected tool calls (#4116).
- Rewrite the agent prompt to describe direct-attachment analysis and
  forbid tool/agent calls. Add a consistency test that extracts the
  prompt's 'available tools' claim and compares it against the
  configured allowlist.

delegate-task (skill-resolver):
- 088693697 filtered per-agent restricted skills at the skill tool and
  builtin agent prompt layers, but delegate-task itself happily injected
  whatever skill name a caller passed. A target agent could be force-fed
  a skill marked agent: oracle just by listing it in load_skills.
- Thread the target agent through resolveSkills and silently filter
  skills whose definition.agent does not include it. Public skills with
  no restriction are unaffected. Regression test pins the bypass.
This commit is contained in:
YeonGyu-Kim
2026-05-22 00:06:56 +09:00
parent 7cce0ad230
commit 11c3da752c
7 changed files with 352 additions and 18 deletions
+53 -3
View File
@@ -1,17 +1,67 @@
import { describe, test, expect } from "bun:test"
import { createAgentToolAllowlist } from "../shared/permission-compat"
import { READ_ENABLED } from "../tools/look-at/look-at-prompt"
import { createMultimodalLookerAgent } from "./multimodal-looker"
function extractAvailableToolClaims(prompt: string): readonly string[] {
const availableToolsLine = prompt
.split("\n")
.find((line) => line.toLowerCase().includes("available tools"))
if (availableToolsLine === undefined) {
return []
}
const tools: string[] = []
for (const match of availableToolsLine.matchAll(/['`]([^'`]+)['`]/g)) {
const toolName = match[1]
if (toolName !== undefined) {
tools.push(toolName)
}
}
return [...new Set(tools)].sort()
}
function allowedToolNames(
toolAllowlist: ReturnType<typeof createAgentToolAllowlist>
): readonly string[] {
return Object.entries(toolAllowlist.permission)
.filter(([toolName, permission]) => toolName !== "*" && permission === "allow")
.map(([toolName]) => toolName)
.sort()
}
function createLookAtRuntimeToolAllowlist(): ReturnType<typeof createAgentToolAllowlist> {
return createAgentToolAllowlist(READ_ENABLED ? ["read"] : [])
}
describe("createMultimodalLookerAgent", () => {
test("prompt explicitly enumerates the agent's available tools to prevent death loop on small VL models", () => {
test("prompt available tool claims match the look_at runtime allowlist", () => {
// given
const agent = createMultimodalLookerAgent("openai/gpt-5-nano")
const runtimeToolAllowlist = createLookAtRuntimeToolAllowlist()
// when
const prompt = typeof agent.prompt === "string" ? agent.prompt : ""
const promptToolClaims = extractAvailableToolClaims(prompt)
const runtimeToolNames = allowedToolNames(runtimeToolAllowlist)
// then
expect(promptToolClaims).toEqual(runtimeToolNames)
})
test("prompt denies tool use to prevent death loop on small VL models", () => {
// given
const agent = createMultimodalLookerAgent("openai/gpt-5-nano")
// when
const prompt = typeof agent.prompt === "string" ? agent.prompt : ""
const normalizedPrompt = prompt.toLowerCase()
// then
expect(prompt).toMatch(/available tools/i)
expect(prompt).toContain("read")
expect(normalizedPrompt).toContain("never")
expect(normalizedPrompt).toContain("tools")
expect(extractAvailableToolClaims(prompt)).toEqual([])
})
test("prompt instructs the agent never to call other tools", () => {
+7 -7
View File
@@ -23,28 +23,28 @@ export function createMultimodalLookerAgent(model: string): AgentConfig {
...restrictions,
prompt: `You interpret media files that cannot be read as plain text.
Your only available tools are 'read' and 'call_omo_agent'. Always use 'read' to load the file first, then analyze the returned content. Never attempt to call any other tool.
During look_at invocations, the file or image is already attached to the message. Analyze the attachment directly. Never call tools, never spawn other agents, and never try to load the file by path.
Your job: examine the attached file and extract ONLY what was requested.
When to use you:
- Media files the Read tool cannot interpret
- Media files that need visual or document interpretation
- Extracting specific information or summaries from documents
- Describing visual content in images or diagrams
- When analyzed/extracted data is needed, not raw file contents
When NOT to use you:
- Source code or plain text files needing exact contents (use Read)
- Files that need editing afterward (need literal content from Read)
- Source code or plain text files needing exact contents
- Files that need editing afterward
- Simple file reading where no interpretation is needed
How you work:
1. Receive a file path and a goal describing what to extract
2. Read and analyze the file deeply
1. Receive an attached file or image and a goal describing what to extract
2. Analyze the attachment deeply
3. Return ONLY the relevant extracted information
4. The main agent never processes the raw file - you save context tokens
For PDFs and documents: Use the Read tool to load the file content first, then extract text, structure, tables, data from specific sections
For PDFs and documents: extract text, structure, tables, and data from specific sections
For images: describe layouts, UI elements, text, diagrams, charts
For diagrams: explain relationships, flows, architecture depicted