extractRetryableSignal returns the raw isRetryable hint from up to 5
nested AI SDK error paths. isRetryableError previously trusted any
true result blindly, which would burn every configured fallback model
in an infinite loop if a provider mis-tagged a 401, 403, or other
non-transient 4xx as retryable.
Honor the signal only when the status code is absent, in the configured
retry_on_errors list, in 5xx, or in {408, 425, 429}. Reject the signal
when the status code is a non-transient 4xx and log the rejection so
operators can debug provider mis-classifications.
Closes pre-publish blocker V8.
The ZAI (Zhipu) provider emits 'Weekly/Monthly Limit Exhausted. Your limit will reset at YYYY-MM-DD HH:MM:SS' when the coding-plan subscription quota is hit. None of the existing quota regex patterns (/quota.?exceeded/, /usage\s+limit/, /exhausted\s+your\s+capacity/, /credit\s+balance.*too\s+low/, etc.) match the 'Limit Exhausted' phrasing, so the runtime-fallback never fires and the user is stuck on the dead model.
Add /limit\s+exhausted/i to both pattern lists that gate fallback dispatch:
- RETRYABLE_ERROR_PATTERNS in constants.ts (text-pattern path used by extractStatusCode + retryable scan)
- classifyErrorType quota_exceeded branch in error-classifier.ts (typed classification path used by isRetryableError)
The pattern is intentionally narrow: it requires the literal token 'Limit' followed by whitespace then 'Exhausted'. It matches the ZAI weekly, monthly, and combined Weekly/Monthly variants but does not collide with unrelated phrases such as 'context limit' or 'rate limit' that already have their own dedicated patterns.
Regression coverage added to quota-error-classifier.regression.test.ts:
- 'Weekly/Monthly Limit Exhausted. Your limit will reset at 2026-05-20 15:43:27' -> quota_exceeded + retryable=true
- 'Weekly Limit Exhausted. Your limit will reset at 2026-05-28 10:30:00' -> quota_exceeded + retryable=true
Verification: 11/11 quota-error-classifier.regression.test.ts pass (was 9 pass + 2 fail pre-fix). Broader runtime-fallback suite goes from 135/196 pass to 137/198 pass (the 61 pre-existing failures are unrelated to this change and reproduce on a clean upstream/dev checkout). bun run typecheck clean.
When the working directory contains a .git folder the OpenCode server
normalises the project root to the git root before persisting messages.
This creates a race: the 429/503/529 error event can fire before the
user's message is committed to storage, so session.messages returns []
and getLastUserRetryParts returns an empty array. The previous code
treated that as a silent no-op (cleared all retry state, Sisyphus stalled).
Fix: when fetchedParts is empty, emit a structured log explaining the
.git-directory race and fall back to a synthetic { type:"text", text:"continue" }
part — matching the pattern already used by autoContinueAfterFallback in
event.ts. The fallback dispatch always proceeds regardless of whether
the messages API can return user parts.
Update four tests that fired two consecutive session.error events relying
on the old silent-stop behaviour: add top-level model fields to the second
error so the awaiting-fallback gate recognises it as coming from the
dispatched fallback model and lets it through normally.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When OpenCode reports the original assistant error after a fallback retry has already been accepted, keep waiting for the pending fallback model instead of clearing the awaiting flag.
This prevents a duplicate stale session.error from advancing the fallback chain and dispatching a second assistant retry prompt.
Addresses cubic-dev-ai P1 finding on #4113 (#4113 review).
The original chain `extractErrorName(error)?.toLowerCase().replace(...)`
is semantically safe — JavaScript optional chaining short-circuits the
ENTIRE access chain when the head returns null/undefined, so when
`extractErrorName` returns undefined the whole expression evaluates to
undefined without ever reaching `.replace()`. Verified empirically via
`const x = undefined; x?.toLowerCase().replace(/_/g, "")` returns
undefined with no crash.
Applying the suggested defensive `?.` before `.replace` anyway, since
it is semantically a no-op and explicit chaining at each hop is easier
for static analyzers to reason about.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses cubic-dev-ai bot review on #4113 (P2): the RESOURCE_EXHAUSTED
and snake_case insufficient_quota fixtures contained quota-shaped
messages that already matched pre-existing message regexes, so the tests
passed even without the new errorName allow-list entry and the
underscore normalization respectively.
Replace both fixture messages with a generic "Request failed." so the
only path to a `quota_exceeded` classification is via the new code:
- RESOURCE_EXHAUSTED: only the new `errorName?.includes("resourceexhausted")`
match on the normalized name can fire.
- insufficient_quota (snake_case): only the new underscore-stripping
normalization can route the name to `insufficientquota` and match the
existing allow-list entry.
The third new test (Google ResourceExhausted message-only) is unchanged
because its message uniquely matches only the new
`/resource.?exhausted/i` pattern and not any existing quota regex.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#3937.
Adds three small classification gaps to `classifyErrorType` so that
quota-exhaustion errors from a wider range of providers trigger
configured fallback chains instead of looping retry attempts:
- Normalize error names by stripping `_` and `-` so snake_case /
SCREAMING_SNAKE_CASE provider names (`insufficient_quota`,
`RESOURCE_EXHAUSTED`, `rate_limit_exceeded`) match the existing
alphanumeric `.includes()` checks.
- Add `resourceexhausted` to the quota error-name allow-list to cover
Google Generative AI's gRPC code 8 / `ResourceExhausted` surface.
- Add `/resource.?exhausted/i` to the quota message-pattern list so the
same error surface is caught when the provider only sets a generic
error name but puts the signal in the message.
Three new regression tests in
`quota-error-classifier.regression.test.ts` cover:
- Google `RESOURCE_EXHAUSTED` (gRPC error name + quota-shaped message)
- Google `ResourceExhausted` message form without HTTP status
- OpenAI snake_case `insufficient_quota` error name
No existing tests were touched; the underscore normalization preserves
all existing `.includes()` matches by rewriting the one underscore-bearing
literal (`ai_loadapikeyerror` → `ailoadapikeyerror`) so previously
matched names still resolve.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the first prompt fails before any durable user message persists,
runtime fallback retry was rebuilding the request from parts alone and
losing the delegated agent system prompt and tool gates. Now it threads
bootstrap.system and bootstrap.tools into the retry body alongside the
captured retry parts, so the retried prompt keeps the same scope as the
initial delegate launch.
Preserve delegated child prompt/bootstrap metadata for early runtime fallback before OpenCode has persisted the first user turn. Bind prompt gate calls to the SDK session receiver and keep completed background task lookup visible across plugin manager instances.
Treat parsed variant as part of runtime-fallback model equivalence so variant-only fallback hops remain distinct while preserving the existing Claude-family alias handling.
Constraint: Oracle verification flagged unresolved PR #3322 review concerns about variant equivalence and remote state
Rejected: Preserve provider identity in equivalence | contradicted the original live-loop fix for equivalent Claude aliases
Confidence: medium
Scope-risk: narrow
Directive: Any future equivalence broadening must prove both live retry-loop behavior and variant/provider semantics with targeted tests before merging
Tested: bun run typecheck
Tested: bun test src/hooks/runtime-fallback/index.test.ts src/hooks/runtime-fallback/error-classifier.test.ts src/plugin/event.model-fallback.test.ts
Not-tested: Full live end-to-end repro across all provider redundancy policies
Prevent runtime fallback from cycling through provider aliases that resolve to the same underlying Claude family model. This keeps retry handling moving toward a genuinely distinct fallback model instead of appearing to fallback while staying on the same effective model.
Constraint: Live retry/fallback bug is in /Users/ravi/Code/personal/oh-my-opencode, while oh-my-openagent contribution work remains isolated to /Users/ravi/Code/forks/oh-my-openagent
Rejected: Change fallback chain precedence (category vs agent) first | lower-confidence root cause than equivalent-model retry
Confidence: high
Scope-risk: narrow
Directive: Keep alias-equivalence logic limited to model families that are intentionally interchangeable for runtime failover, and expand with targeted tests before broadening provider-family collapsing
Tested: bun run typecheck
Tested: bun test src/hooks/runtime-fallback/index.test.ts src/hooks/runtime-fallback/error-classifier.test.ts src/plugin/event.model-fallback.test.ts
Not-tested: Full live end-to-end session repro against external provider outages
Addresses two issues identified by cubic on PR #3952.
1. Watchdog cancellation was too narrow — only `text`/`reasoning` parts
counted as progress, so a subagent that immediately ran tools
(Read/Bash/Edit) emitted `tool`/`tool_use`/`tool_result`/`tool-call`/
`step-start` parts that the watchdog ignored, risking a false fire
on actively-working subagents. Broaden to: any assistant part of any
known type counts as progress (the model has started responding,
whether or not visible text has arrived yet). `info.error` and
`info.finish` continue to cancel.
2. Test timing margins were tight (15ms pre-cancel against a 40ms
timer), risking CI flakiness on loaded runners. Bumped to a 100ms
threshold with a 40ms pre-cancel window and a 250ms post-fire wait,
giving a 60ms margin before the timer fires and ~2.5x the threshold
after — robust against scheduler delay.
Refactor for testability: extracted the OpenCode-event→watchdog-signal
translation out of `hook.ts` into an exported `observeEventForWatchdog`
helper on the watchdog module. This let me add direct unit tests for
every part-type case (text, reasoning, tool, tool_use, tool_result,
tool-call, step-start, file) plus the error/finish/empty-parts branches
without spinning up the full hook. Net diff: hook.ts shrinks, watchdog
module gains a small pure function with parametrised coverage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a subagent is dispatched to a provider and the underlying SDK
enters a silent internal retry loop on a 429/quota error, no error
event is ever emitted back to OpenCode. The runtime-fallback hook —
which is fully reactive (listens to message.updated/session.error/
session.status) — has nothing to react to and never dispatches the
configured fallback. The subagent sits in `retry` status until the
parent's 30-minute poll timeout (DEFAULT_POLL_TIMEOUT_MS) gives up,
during which the parent's pending task tool call shows "waiting for
subagent" with no indication of failure.
This change adds a first-prompt watchdog that synthesises the missing
error-event trigger:
- Armed when a user message lands in a subagent session
(membership check via `subagentSessions`).
- Cancelled on the first sign of progress: any assistant message
with text/reasoning content, finish field, or an error field (any
of which is something the existing handlers will deal with).
- Cancelled on session terminal events (idle/stop/deleted/error).
- On fire (90s default): aborts the in-flight request and routes
into the existing dispatchFallbackRetry path — the same code that
runs when a session.error arrives. No new fallback mechanism.
Design choices:
- Dispatch fallback, do not abort the subagent outright. Network
loss looks identical to a stuck retry from the hook's vantage
point; with fallback-dispatch behaviour, network loss degrades
to today's baseline (both attempts fail, 30-min outer timeout
still ends things) rather than destructively aborting work.
- Scope strictly to subagents. Parent/user sessions can legitimately
take 90s+ to produce the first token; subagent dispatches in
practice produce first content much faster, so a 90s ceiling is
safe.
- Threshold is tunable via the third arg to createFirstPromptWatchdog;
DEFAULT_FIRST_PROMPT_WATCHDOG_MS = 90_000 in constants.ts.
Also adds a diagnostic log in session-status-handler when a
`session.status: retry` event arrives whose message does not match
RETRYABLE_ERROR_PATTERNS. This is the hook's other silent-return
spot for retry events; logging the raw retry message will let us
extend the patterns next time we hit a provider whose phrasing
we don't yet match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep prompt reservations briefly after successful dispatch so rapid idle/message/error transitions cannot inject the same follow-up twice.
Route all production session prompt calls through the shared gate, restore skipped background resume state, release holds after abort/recovery paths, and preserve Ralph/ULW loop state when a dispatch is deferred.
Add regression coverage for session routing, static prompt route auditing, team-mode live messaging, model suggestion retries, call-omo-agent reuse, background parent wakes, runtime fallback, compaction recovery, Atlas, and Ralph/ULW loops.
When runtime-fallback aborts an in-flight request to swap in a fallback
model, opencode emits session.error{isAbort:true} as a consequence. The
existing event handler treated that as a user cancellation and called
resetRetryState — wiping attemptCount. Every subsequent provider
auto-retry signal then started over at attempt:1, never reaching
max_fallback_attempts, producing an infinite retry loop firing a new
fallback every ~2 seconds.
The bug only surfaces when the configured fallback target itself
silently fails (e.g. github-copilot quota exhausted): the original
model keeps re-emitting retry signals, our handler keeps "fixing"
them, the counter never advances. Reproducible on upstream/dev HEAD
(5ffbe0e24e).
Fix:
- New `internallyAbortedSessions: Set<string>` on HookDeps tracks
sessions whose abort we triggered ourselves.
- abortSessionRequest in auto-retry.ts adds the session to the set
when called with one of our internal sources:
"session.status.retry-signal", "message.updated.retry-signal",
"session.timeout". The "session.stop" source (user-initiated) is
intentionally NOT marked — that path must still wipe state.
- handleSessionError in event-handler.ts checks the set before the
cancellation branch. If the session is marked, consume the flag
(delete it so a later user-abort still gets the reset) and skip
resetRetryState. The state's attemptCount is preserved, so the
next iteration progresses 1→2→3→... until max_fallback_attempts.
- dispose() clears the new set alongside the other per-session maps.
Tests: 3 new event-handler integration tests cover the fix
(internal-abort preserves state, external-abort still resets,
consecutive internal-abort cycles advance attemptCount). Existing
tests pass: 7/7 on event-handler. Pre-existing 2 dispose-test flakes
on the full runtime-fallback suite were verified to exist on
upstream/dev without this patch — unrelated.
bun run build: pass. bun run typecheck: pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sync the PR branch with the newest dev branch and resolve the new import-level conflicts in background-agent manager and runtime-fallback tests. Preserve both the delegated bootstrap coverage from this branch and the newer upstream test utilities and runtime wiring changes, then re-verify the affected delegated fallback suites and typecheck.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Handle OpenCode session events that carry the session ID under properties.info.id or properties.info.sessionID so background tasks and continuation hooks do not miss idle/error/delete events.
Add regression coverage for nested session.idle events completing background tasks and waking continuation hooks.
Sync the PR branch with the latest dev branch and resolve the remaining conflict in sync-task.test.ts while preserving both the new upstream poll-recovery coverage and this branch's delegated bootstrap cleanup and isolation coverage. Re-verified the affected delegated fallback suites and typecheck after the merge resolution.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
When a subagent session (e.g. Momus on GPT) hits a quota/usage-limit
error and the agent's category has no `fallback_models` configured, the
runtime-fallback hook previously returned silently. The OpenCode session
stayed in `retry` status indefinitely while the SDK kept hitting the
limit, the sync-task poller treated `retry` as active work, and the
parent's pending `task` tool call never resolved — leaving a stuck
"waiting for subagent" indicator in the parent conversation.
Narrow fix: at the `fallbackModels.length === 0` exit point, if the
session is a known subagent AND the error classifies as
`quota_exceeded`, abort the subagent session. The existing
`getTerminalSessionError` path in `sync-session-poller.ts` then surfaces
the error via the parent's tool result, which is the persistent surface
the user is already watching.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ToToKr