Commit Graph

100 Commits

Author SHA1 Message Date
codivedev b4d4d30fa8 fix: use display name in runtime-fallback retry 2026-03-27 13:39:50 +01:00
YeonGyu-Kim e6d0484e57 Merge pull request #2710 from MoerAI/fix/rate-limit-hang
fix(runtime-fallback): detect bare 429 rate-limit signals (fixes #2677)
2026-03-26 08:53:41 +09:00
YeonGyu-Kim abd62472cf Merge pull request #2752 from MoerAI/fix/quota-error-fallback-detection
fix(runtime-fallback): detect prettified quota errors without HTTP status codes (fixes #2747)
2026-03-26 08:50:58 +09:00
Ravi Tharuma a77a16c494 feat(config): support object-style fallback_models with per-model settings
Add support for object-style entries in fallback_models arrays, enabling
per-model configuration of variant, reasoningEffort, temperature, top_p,
maxTokens, and thinking settings.

- Zod schema for FallbackModelObject with full validation
- normalizeFallbackModels() and flattenToFallbackModelStrings() utilities
- Provider-agnostic model resolution pipeline with fallback chain
- Session prompt params state management
- Fallback chain construction with prefix-match lookup
- Integration across delegate-task, background-agent, and plugin layers
2026-03-25 11:04:49 +01:00
YeonGyu-Kim 7761e48dca Merge pull request #2592 from MoerAI/fix/gemini-quota-fallback
fix(runtime-fallback): detect Gemini quota errors in session.status retry events (fixes #2454)
2026-03-25 18:14:21 +09:00
MoerAI 44fb114370 fix(runtime-fallback): rename misleading test to match actual behavior
The test name claimed it exercised RETRYABLE_ERROR_PATTERNS directly,
but classifyErrorType actually matches 'payment required' via the
quota_exceeded path first. Rename to 'detects payment required errors
as retryable' to accurately describe end-to-end behavior.
2026-03-25 16:58:49 +09:00
MoerAI f16d55ad95 fix: add errorName-based quota detection and strengthen test coverage 2026-03-23 15:19:09 +09:00
MoerAI 62d2704009 fix(runtime-fallback): detect prettified quota errors without HTTP status codes (fixes #2747) 2026-03-23 10:34:22 +09:00
MoerAI 3773e370ec fix(runtime-fallback): detect bare 429 rate-limit signals (fixes #2677) 2026-03-20 11:00:00 +09:00
YeonGyu-Kim c5c7ba4eed perf: pre-compile regex patterns and optimize hot-path string operations
- error-classifier: pre-compile default retry pattern regex
- think-mode/detector: combine multilingual patterns into single regex
- parser: skip redundant toLowerCase on pre-lowered keywords
- edit-operations: use fast arraysEqual instead of JSON comparison
- hash-computation: optimize streaming line extraction with index tracking
2026-03-18 14:19:23 +09:00
YeonGyu-Kim 9ca259dcdc fix(runtime-fallback): preserve agent variant and reasoningEffort on model fallback (fixes #2621)
When runtime fallback switches to a different model, the agent's
configured variant and reasoningEffort were lost because
buildRetryModelPayload only extracted variant from the fallback
model string itself.

Now buildRetryModelPayload accepts optional agentSettings and uses
the agent's variant as fallback when the model string doesn't
include one. reasoningEffort is also passed through.

🤖 Generated with assistance of [OhMyOpenCode](https://github.com/code-yeongyu/oh-my-opencode)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 12:11:01 +09:00
Ravi Tharuma de66f1f397 fix(runtime-fallback): prefer numeric status codes over non-numeric in extraction chain
The nullish-coalescing chain could stop at a non-numeric value (e.g.
status: "error"), preventing deeper nested numeric statusCode values
from being reached. Switch to Array.find() with a type guard to always
select the first numeric value.

Adds 11 tests for extractStatusCode covering: top-level, nested
(data/error/cause), non-numeric skip, fallback to regex, and
precedence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 13:51:23 +01:00
Ravi Tharuma 24a0f7b032 fix(runtime-fallback): extract status code from nested AI SDK errors
AI SDK wraps HTTP status codes inside error.error.statusCode (e.g., AI_APICallError). The current extractStatusCode only checks the top level, missing these nested codes.

This caused runtime-fallback to skip retryable errors like 400, 500, 504 because it couldn't find the status code.

Fixes #2617
2026-03-16 13:04:14 +01:00
MoerAI 11d942f3a2 fix(runtime-fallback): detect Gemini quota errors in session.status retry events
When Gemini returns a quota exhausted error, OpenCode auto-retries and
fires session.status with type='retry'. The extractAutoRetrySignal
function requires BOTH 'retrying in' text AND a quota pattern to match,
but some providers (like Gemini) include only the error text in the
retry message without the 'retrying in' phrase.

Since status.type='retry' already confirms this is a retry event, the
fix adds a fallback check: if extractAutoRetrySignal fails, check the
message directly against RETRYABLE_ERROR_PATTERNS. This ensures quota
errors like 'exhausted your capacity' trigger the fallback chain even
when the retry message format differs from expected.

Fixes #2454
2026-03-16 11:08:14 +09:00
YeonGyu-Kim 1ad5db4e8b fix(runtime-fallback): advance session.status fallback chain
Allow provider cooldown events to override a pending fallback retry so runtime fallback can keep progressing instead of stalling on the same model.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-16 10:38:55 +09:00
YeonGyu-Kim 9bce6314b1 fix(runtime-fallback): scope visible-assistant check to current turn and cleanup retry dedupe keys 2026-03-13 10:54:47 +09:00
YeonGyu-Kim abc4b2a6a4 fix(runtime-fallback): remove committed rebase conflict markers
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:49:46 +09:00
YeonGyu-Kim d8da2f1ad6 fix(runtime-fallback): clear retry keys on failed session bootstrap
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:39:30 +09:00
YeonGyu-Kim 62a905b690 fix(runtime-fallback): reuse normalized messages for visible assistant checks
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:39:30 +09:00
YeonGyu-Kim 79fb746a1c fix(runtime-fallback): resolve agents from normalized session messages
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:39:30 +09:00
YeonGyu-Kim fcd4fa5164 fix(runtime-fallback): normalize retry part message extraction
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:39:30 +09:00
YeonGyu-Kim 6a4a3322c1 fix(runtime-fallback): add session messages extractor
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:39:30 +09:00
YeonGyu-Kim 3caa3fcc3d fix: address Cubic findings for runtime fallback child sessions
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-03-12 13:39:30 +09:00
YeonGyu-Kim ba86ef0eea fix: enable runtime fallback for delegated child sessions (#2357) 2026-03-12 13:39:04 +09:00
YeonGyu-Kim 2d2ca863f1 fix(runtime-fallback): clear monitoring interval on dispose
setInterval for model availability monitoring was never cleared,
keeping the hook alive indefinitely with no dispose mechanism.

- Add dispose() method to RuntimeFallbackHook that clears interval
- Track intervalId in hook state for cleanup
- Export dispose in hook return type

Tests: 3 pass, 10 expects
2026-03-12 01:37:02 +09:00
Ravi Tharuma c598afa521 Address PR review follow-ups for retry status handling 2026-03-09 12:43:30 +09:00
Ravi Tharuma 38c925697b Respect per-agent fallback chains in runtime fallback 2026-03-09 12:43:01 +09:00
Ravi Tharuma 4300f60aaf Detect runtime retry signals from assistant text parts 2026-03-09 12:43:01 +09:00
Ravi Tharuma eab5be666d Fix cooldown fallback switching across model/runtime fallback hooks 2026-03-09 12:43:01 +09:00
YeonGyu-Kim fade6740ae chore: update GPT-5.2 references to GPT-5.4
Align runtime defaults, tests, docs, and generated artifacts with the newer GPT-5.4 baseline. Keep think-mode and prompt-routing expectations consistent after the model version bump.
2026-03-07 05:46:05 +09:00
YeonGyu-Kim d425f9bb80 fix(models): update Gemini 3 to 3.1 Pro and add Kimi to writing category fallback
Closes #2065

Closes #1968
2026-02-26 21:01:26 +09:00
YeonGyu-Kim 976798d0e3 feat(config): disable runtime_fallback by default (opt-in) 2026-02-22 01:54:34 +09:00
YeonGyu-Kim fe415319e5 fix: resolve publish blockers for v3.7.4→v3.8.0 release
- Fix #1991 crash: optional chaining for task-history sessionID access
- Fix #1992 think-mode: add antigravity entries to HIGH_VARIANT_MAP
- Fix #1949 Copilot premium misattribution: use createInternalAgentTextPart
- Fix #1982 load_skills: pass directory to discoverSkills for project-level skills
- Fix command priority: sort scopePriority before .find(), project-first return
- Fix Google provider transform: apply in userFallbackModels path
- Fix ralph-loop TUI: optional chaining for event handler
- Fix runtime-fallback: unify dual fallback engines, remove HTTP 400 from retry,
  fix pendingFallbackModel stuck state, add priority gate to skip model-fallback
  when runtime-fallback is active
- Fix Prometheus task system: exempt from todowrite/todoread deny
- Fix background_output: default full_session to true
- Remove orphan hooks: hashline-edit-diff-enhancer (redundant with hashline_edit
  built-in diff), task-reminder (dead code)
- Remove orphan config entries: 3 stale hook names from Zod schema
- Fix disabled_hooks schema: accept arbitrary strings for forward compatibility
- Register json-error-recovery hook in tool-guard pipeline
- Add disabled_hooks gating for question-label-truncator, task-resume-info,
  claude-code-hooks
- Update test expectations to match new behavior
2026-02-21 16:24:18 +09:00
YeonGyu-Kim 8623f58a38 fix: resolve 5 deployment blockers (runtime-fallback race, hashline legacy, tmux spawn, db open)
- runtime-fallback: guard session.error with sessionRetryInFlight to prevent
  double-advance during active retry; expand session.stop abort to include
  sessionAwaitingFallbackResult; remove premature pendingFallbackModel clearing
  from auto-retry finally block
- hashline-edit: add HASHLINE_LEGACY_REF_PATTERN for backward-compatible
  LINE:HEX dual-parse in parseLineRef and normalizeLineRef
- tmux-subagent: defer session on null queryWindowState; unconditionally
  re-queue deferred session on spawn failure (not just close+spawn)
- ultrawork-db: wrap new Database(dbPath) in try/catch to handle corrupted DB
- event: add try/catch guards around model-fallback logic in message.updated,
  session.status, and session.error handlers
2026-02-21 05:59:30 +09:00
IYODA Atsushi fcaaa11a06 fix(runtime-fallback): detect type:error message parts for fallback progression 2026-02-21 02:42:20 +09:00
Youngbin Kim b6456faea8 refactor(runtime-fallback): decompose index.ts into focused modules
Split 1021-line index.ts into 10 focused modules per project conventions.

New structure:

- error-classifier.ts: error analysis with dynamic status code extraction

- agent-resolver.ts: agent detection utilities

- fallback-state.ts: state management and cooldown logic

- fallback-models.ts: model resolution from config

- auto-retry.ts: retry helpers with mutual recursion support

- event-handler.ts: session lifecycle events

- message-update-handler.ts: message.updated event handling

- chat-message-handler.ts: chat message interception

- hook.ts: main factory with proper cleanup

- types.ts: updated with HookDeps interface

- index.ts: 2-line barrel re-export

Embedded fixes:

- Fix setInterval leak with .unref()

- Replace require() with ESM import

- Add log warning on invalid model format

- Update sessionLastAccess on normal traffic

- Make extractStatusCode dynamic from config

- Remove unused SessionErrorInfo type

All 61 tests pass without modification.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-02-21 02:42:12 +09:00
Youngbin Kim 68f5d982fc feat(runtime-fallback): add timeout toggle for quota retry detection
Make provider auto-retry signal detection respect timeout_seconds setting:
- When timeout_seconds=0, disable quota-based fallback escalation
- Only treat auto-retry signals as errors when timeout is enabled
- Add test to verify behavior when timeout_seconds is disabled
- Update documentation to explain timeout_seconds=0 behavior

This allows users to disable timeout-based fallbacks while keeping
error-based fallback functionality intact.
2026-02-21 02:40:47 +09:00
Youngbin Kim 8b2ae957e5 feat(runtime-fallback): generalize provider auto-retry signal detection
Refactor retry signal detection to be provider-agnostic:
- Replace hardcoded Copilot/OpenAI checks with generic pattern matching
- Detect any provider message containing limit/quota keywords + [retrying in X]
- Add OpenAI pattern: 'usage limit has been reached [retrying in X]'
- Update logging to use generic 'provider' instead of specific names
- Add 'usage limit has been reached' to RETRYABLE_ERROR_PATTERNS

This enables fallback escalation for any provider that signals automatic
retries due to quota/rate limits, not just Copilot and OpenAI.

Closes PR discussion: generalize retry pattern detection
2026-02-21 02:40:47 +09:00
Youngbin Kim 6a97f00a22 feat(runtime-fallback): add configurable session timeout controls 2026-02-21 02:40:01 +09:00
Youngbin Kim ff230df47c fix(runtime-fallback): harden fallback progression and success detection 2026-02-21 02:40:01 +09:00
Youngbin Kim 5a406cab9e refactor(runtime-fallback): extract auto-retry helper and fix provider constraint inconsistency
- Extract duplicated auto-retry logic (~40 lines each) from session.error and
  message.updated handlers into shared autoRetryWithFallback() helper
- Fix userFallbackModels path in model-resolution-pipeline to respect
  constraints.connectedProviders parameter instead of reading cache directly,
  matching the behavior of categoryDefaultModel and fallbackChain paths
2026-02-21 02:40:01 +09:00
Youngbin Kim fbafb8cf67 fix(runtime-fallback): 9 critical bug fixes for auto-retry, agent preservation, and model override
Bug fixes:
1. extractStatusCode: handle nested data.statusCode (Anthropic error structure)
2. Error regex: relax credit.*balance.*too.*low pattern for multi-char gaps
3. Zod schema: bump max_fallback_attempts from 10 to 20 (config rejected silently)
4. getFallbackModelsForSession: fallback to sisyphus/any agent when session.error lacks agent
5. Model detection: derive model from agent config when session.error lacks model info
6. Auto-retry: resend last user message with fallback model via promptAsync
7. Persistent fallback: override model on every chat.message (not just pendingFallbackModel)
8. Manual model change: detect UI model changes and reset fallback state
9. Agent preservation: include agent in promptAsync body to prevent defaulting to sisyphus

Additional:
- Add sessionRetryInFlight guard to prevent double-retries
- Add resolveAgentForSession with 3-tier resolution (event → session memory → session ID)
- Add normalizeAgentName for display names like "Prometheus (Planner)" → "prometheus"
- Add resolveAgentForSessionFromContext to fetch agent from session messages
- Move AGENT_NAMES and agentPattern to module scope for reuse
- Register runtime-fallback hooks in event.ts and chat-message.ts
- Remove diagnostic debug logging from isRetryableError
- Add 400 to default retry_on_errors and credit/balance patterns to RETRYABLE_ERROR_PATTERNS
2026-02-21 02:39:41 +09:00
youming.tang 708b9ce9ff fix(runtime-fallback): sort agent names by length to fix hyphenated agent detection
The \b word boundary regex treats '-' as a boundary, causing
'sisyphus-junior-session-123' to incorrectly match 'sisyphus'
instead of 'sisyphus-junior'.

Sorting agent names by length (descending) ensures longer names
are matched first, fixing the hyphenated agent detection issue.

Fixes cubic-dev-ai review issue #8
2026-02-21 02:38:17 +09:00
um1ng d9072b4a98 fix(runtime-fallback): address cubic AI review issues
- Add normalizeFallbackModels helper to centralize string/array normalization (P3)
- Export RuntimeFallbackConfig and FallbackModels types from config/index.ts
- Fix agent detection regex to use word boundaries for sessionID matching
- Improve tests to verify actual fallback switching logic (not just log paths)
- Add SessionCategoryRegistry cleanup in executeSyncTask on completion/error (P2)
- All 24 runtime-fallback tests pass, 115 delegate-task tests pass
2026-02-21 02:37:57 +09:00
youming.tang 067c8010be fix: resolve merge conflicts in PR #1408
- Fix bun.lock version conflicts (3.3.1 -> 3.3.2)
- Remove Git conflict markers from docs/configurations.md
- Remove duplicate normalizeFallbackModels, import from shared module
2026-02-21 02:35:03 +09:00
um1ng 8873896432 fix(runtime-fallback): use precise regex patterns for status code matching
Replace word-boundary regex with stricter patterns that match

status codes only at start/end of string or surrounded by whitespace.

Prevents false matches like '1429' or '4290'.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-02-21 02:33:49 +09:00
youming.tang d947743932 fix(runtime-fallback): per-model cooldown and stricter retry patterns 2026-02-21 02:30:55 +09:00
Ultrawork Bot 7aafa13b21 feat(fallback_models): complete init-time and runtime integration
Implement full fallback_models support across all integration points:

1. Model Resolution Pipeline (src/shared/model-resolution-pipeline.ts)
   - Add userFallbackModels to ModelResolutionRequest
   - Process user fallback_models before hardcoded fallback chain
   - Support both connected provider and availability checking modes

2. Agent Utils (src/agents/utils.ts)
   - Update applyModelResolution to accept userFallbackModels
   - Inject fallback_models for all builtin agents (sisyphus, oracle, etc.)
   - Support both single string and array formats

3. Model Resolver (src/shared/model-resolver.ts)
   - Add userFallbackModels to ExtendedModelResolutionInput type
   - Pass through to resolveModelPipeline

4. Delegate Task Executor (src/tools/delegate-task/executor.ts)
   - Extract category fallback_models configuration
   - Pass to model resolution pipeline
   - Register session category for runtime-fallback hook

5. Session Category Registry (src/shared/session-category-registry.ts)
   - New module: maps sessionID -> category
   - Used by runtime-fallback to lookup category fallback_models
   - Auto-cleanup support

6. Runtime Fallback Hook (src/hooks/runtime-fallback/index.ts)
   - Check SessionCategoryRegistry first for category fallback_models
   - Fallback to agent-level configuration
   - Import and use SessionCategoryRegistry

Test Results:
- runtime-fallback: 24/24 tests passing
- model-resolver: 46/46 tests passing

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-02-21 02:30:01 +09:00
Rebase Bot 6dc1aff698 fix(runtime-fallback): add Category support and expand test coverage
- Add Category-level fallback_models support in getFallbackModelsForSession()
  - Try agent-level fallback_models first
  - Then try agent's category fallback_models
  - Support all builtin agents including hephaestus, sisyphus-junior, build, plan

- Expand agent name recognition regex to include:
  - hephaestus, sisyphus-junior, build, plan, multimodal-looker

- Add comprehensive test coverage (6 new tests, total 24):
  - Model switching via chat.message hook
  - Agent-level fallback_models configuration
  - SessionID agent pattern detection
  - Cooldown mechanism validation
  - Max attempts limit enforcement

All 24 tests passing

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-02-21 02:28:27 +09:00
Rebase Bot 632570f7ec feat(config): add runtime_fallback and fallback_models schema
Add configuration schemas for runtime model fallback feature:
- RuntimeFallbackConfigSchema with enabled, retry_on_errors,
  max_fallback_attempts, cooldown_seconds, notify_on_fallback
- FallbackModelsSchema for init-time fallback model selection
- Add fallback_models to AgentOverrideConfigSchema and CategoryConfigSchema
- Export types and schemas from config/index.ts

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-02-21 02:28:27 +09:00