feat(model-settings-compat): add variant/reasoningEffort compatibility resolver
- Registry-based model family detection (provider-agnostic) - Variant and reasoningEffort ladder downgrade logic - Three-tier resolution: metadata override → family heuristic → unknown drop - Comprehensive test suite covering all model families
This commit is contained in:
Ravi Tharuma
@@ -0,0 +1,86 @@
|
||||
# Model Settings Compatibility Resolver Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Centralize compatibility handling for `variant` and `reasoningEffort` so an already-selected model receives the best valid settings for that exact model.
|
||||
|
||||
**Architecture:** Introduce a pure shared resolver in `src/shared/` that computes compatible settings and records downgrades/removals. Integrate it first in `chat.params`, then keep Claude-specific effort logic as a thin layer rather than a special-case policy owner.
|
||||
|
||||
**Tech Stack:** TypeScript, Bun test, existing shared model normalization/utilities, OpenCode plugin `chat.params` path.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Create the pure compatibility resolver
|
||||
|
||||
**Files:**
|
||||
- Create: `src/shared/model-settings-compatibility.ts`
|
||||
- Create: `src/shared/model-settings-compatibility.test.ts`
|
||||
- Modify: `src/shared/index.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for exact keep behavior**
|
||||
- [ ] **Step 2: Write failing tests for downgrade behavior (`max` -> `high`, `xhigh` -> `high` where needed)**
|
||||
- [ ] **Step 3: Write failing tests for unsupported-value removal**
|
||||
- [ ] **Step 4: Write failing tests for model-family distinctions (Opus vs Sonnet/Haiku, GPT-family variants)**
|
||||
- [ ] **Step 5: Implement the pure resolver with explicit capability ladders**
|
||||
- [ ] **Step 6: Export the resolver from `src/shared/index.ts`**
|
||||
- [ ] **Step 7: Run `bun test src/shared/model-settings-compatibility.test.ts`**
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
### Task 2: Integrate resolver into chat.params
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/plugin/chat-params.ts`
|
||||
- Modify: `src/plugin/chat-params.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing tests showing `chat.params` applies resolver output to runtime settings**
|
||||
- [ ] **Step 2: Ensure tests cover both `variant` and `reasoningEffort` decisions**
|
||||
- [ ] **Step 3: Update `chat-params.ts` to call the shared resolver before hook-specific adjustments**
|
||||
- [ ] **Step 4: Preserve existing prompt-param-store merging behavior**
|
||||
- [ ] **Step 5: Run `bun test src/plugin/chat-params.test.ts`**
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
### Task 3: Re-scope anthropic-effort around the resolver
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/hooks/anthropic-effort/hook.ts`
|
||||
- Modify: `src/hooks/anthropic-effort/index.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing tests that codify the intended remaining Anthropic-specific behavior after centralization**
|
||||
- [ ] **Step 2: Reduce `anthropic-effort` to Claude/Anthropic-specific effort injection where still needed**
|
||||
- [ ] **Step 3: Remove duplicated compatibility policy from the hook if the shared resolver now owns it**
|
||||
- [ ] **Step 4: Run `bun test src/hooks/anthropic-effort/index.test.ts`**
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
### Task 4: Add integration/regression coverage across real request paths
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/plugin/chat-params.test.ts`
|
||||
- Modify: `src/hooks/anthropic-effort/index.test.ts`
|
||||
- Add tests only where needed in nearby suites
|
||||
|
||||
- [ ] **Step 1: Add regression test for non-Opus Claude with `variant=max` resolving to compatible settings without ad hoc path-only logic**
|
||||
- [ ] **Step 2: Add regression test for GPT-style `reasoningEffort` compatibility**
|
||||
- [ ] **Step 3: Add regression test showing supported values remain unchanged**
|
||||
- [ ] **Step 4: Run the focused test set**
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
### Task 5: Verify full quality bar
|
||||
|
||||
**Files:**
|
||||
- No intended code changes
|
||||
|
||||
- [ ] **Step 1: Run `bun run typecheck`**
|
||||
- [ ] **Step 2: Run a focused suite for the touched files**
|
||||
- [ ] **Step 3: If clean, run `bun test`**
|
||||
- [ ] **Step 4: Review diff for accidental scope creep**
|
||||
- [ ] **Step 5: Commit any final cleanup**
|
||||
|
||||
### Task 6: Prepare PR metadata
|
||||
|
||||
**Files:**
|
||||
- No repo file change required unless docs are updated further
|
||||
|
||||
- [ ] **Step 1: Write a human summary explaining this is settings compatibility, not model fallback**
|
||||
- [ ] **Step 2: Document scope: Phase 1 covers `variant` and `reasoningEffort` only**
|
||||
- [ ] **Step 3: Document explicit non-goals: no model switching, no automatic upscaling in Phase 1**
|
||||
- [ ] **Step 4: Request review**
|
||||
@@ -0,0 +1,164 @@
|
||||
# Model Settings Compatibility Resolver Design
|
||||
|
||||
## Goal
|
||||
|
||||
Introduce a central resolver that takes an already-selected model and a set of desired model settings, then returns the best compatible configuration for that exact model.
|
||||
|
||||
This is explicitly separate from model fallback.
|
||||
|
||||
## Problem
|
||||
|
||||
Today, logic for `variant` and `reasoningEffort` compatibility is scattered across multiple places:
|
||||
- `hooks/anthropic-effort`
|
||||
- `plugin/chat-params`
|
||||
- agent/category/fallback config layers
|
||||
- delegate/background prompt plumbing
|
||||
|
||||
That creates inconsistent behavior:
|
||||
- some paths clamp unsupported levels
|
||||
- some paths pass them through unchanged
|
||||
- some paths silently drop them
|
||||
- some paths use model-family-specific assumptions that do not generalize
|
||||
|
||||
The result is brittle request behavior even when the chosen model itself is valid.
|
||||
|
||||
## Scope
|
||||
|
||||
Phase 1 covers only:
|
||||
- `variant`
|
||||
- `reasoningEffort`
|
||||
|
||||
Out of scope for Phase 1:
|
||||
- model fallback itself
|
||||
- `thinking`
|
||||
- `maxTokens`
|
||||
- `temperature`
|
||||
- `top_p`
|
||||
- automatic upward remapping of settings
|
||||
|
||||
## Desired behavior
|
||||
|
||||
Given a fixed model and desired settings:
|
||||
1. If a desired value is supported, keep it.
|
||||
2. If not supported, downgrade to the nearest lower compatible value.
|
||||
3. If no compatible value exists, drop the field.
|
||||
4. Do not switch models.
|
||||
5. Do not automatically upgrade settings in Phase 1.
|
||||
|
||||
## Architecture
|
||||
|
||||
Add a central module:
|
||||
- `src/shared/model-settings-compatibility.ts`
|
||||
|
||||
Core API:
|
||||
|
||||
```ts
|
||||
type DesiredModelSettings = {
|
||||
variant?: string
|
||||
reasoningEffort?: string
|
||||
}
|
||||
|
||||
type ModelSettingsCompatibilityInput = {
|
||||
providerID: string
|
||||
modelID: string
|
||||
desired: DesiredModelSettings
|
||||
}
|
||||
|
||||
type ModelSettingsCompatibilityChange = {
|
||||
field: "variant" | "reasoningEffort"
|
||||
from: string
|
||||
to?: string
|
||||
reason: string
|
||||
}
|
||||
|
||||
type ModelSettingsCompatibilityResult = {
|
||||
variant?: string
|
||||
reasoningEffort?: string
|
||||
changes: ModelSettingsCompatibilityChange[]
|
||||
}
|
||||
```
|
||||
|
||||
## Compatibility model
|
||||
|
||||
Phase 1 should be **metadata-first where the platform exposes reliable capability data**, and only fall back to family-based rules when that metadata is absent.
|
||||
|
||||
### Variant compatibility
|
||||
|
||||
Preferred source of truth:
|
||||
- OpenCode/provider model metadata (`variants`)
|
||||
|
||||
Fallback when metadata is unavailable:
|
||||
- family-based ladders
|
||||
|
||||
Examples of fallback ladders:
|
||||
- Claude Opus family: `low`, `medium`, `high`, `max`
|
||||
- Claude Sonnet/Haiku family: `low`, `medium`, `high`
|
||||
- OpenAI GPT family: conservative family fallback only when metadata is missing
|
||||
- Unknown family: drop unsupported values conservatively
|
||||
|
||||
### Reasoning effort compatibility
|
||||
|
||||
Current Phase 1 source of truth:
|
||||
- conservative model/provider family heuristics
|
||||
|
||||
Reason:
|
||||
- the currently available OpenCode SDK/provider metadata exposes model `variants`, but does not expose an equivalent per-model capability list for `reasoningEffort` levels
|
||||
|
||||
Examples:
|
||||
- GPT/OpenAI-style models: `low`, `medium`, `high`, `xhigh` where supported by family heuristics
|
||||
- Claude family via current OpenCode path: treat `reasoningEffort` as unsupported in Phase 1 and remove it
|
||||
|
||||
The resolver should remain pure model/settings logic only. Transport restrictions remain the responsibility of the request-building path.
|
||||
|
||||
## Separation of concerns
|
||||
|
||||
This design intentionally separates:
|
||||
- model selection (`resolveModel...`, fallback chains)
|
||||
- settings compatibility (this resolver)
|
||||
- request transport compatibility (`chat.params`, prompt body constraints)
|
||||
|
||||
That keeps responsibilities clear:
|
||||
- choose model first
|
||||
- normalize settings second
|
||||
- build request third
|
||||
|
||||
## First integration point
|
||||
|
||||
Phase 1 should first integrate into `chat.params`.
|
||||
|
||||
Why:
|
||||
- it is already the centralized path for request-time tuning
|
||||
- it can influence provider-facing options without leaking unsupported fields into prompt payload bodies
|
||||
- it avoids trying to patch every prompt constructor at once
|
||||
|
||||
## Rollout plan
|
||||
|
||||
### Phase 1
|
||||
- add resolver module and tests
|
||||
- integrate into `chat.params`
|
||||
- migrate `anthropic-effort` to either use the resolver or become a thin Claude-specific supplement around it
|
||||
|
||||
### Phase 2
|
||||
- expand to `thinking`, `maxTokens`, `temperature`, `top_p`
|
||||
- formalize request-path capability tables if needed
|
||||
|
||||
### Phase 3
|
||||
- centralize all variant/reasoning normalization away from scattered hooks and ad hoc callers
|
||||
|
||||
## Risks
|
||||
|
||||
- Overfitting family rules to current model naming conventions
|
||||
- Accidentally changing request semantics on paths that currently rely on implicit behavior
|
||||
- Mixing provider transport limitations with model capability logic
|
||||
|
||||
## Mitigations
|
||||
|
||||
- Keep resolver pure and narrowly scoped in Phase 1
|
||||
- Add explicit regression tests for keep/downgrade/drop decisions
|
||||
- Integrate at one central point first (`chat.params`)
|
||||
- Preserve existing behavior where desired values are already valid
|
||||
|
||||
## Recommendation
|
||||
|
||||
Proceed with the central resolver as a new, isolated implementation in a dedicated branch/worktree.
|
||||
This is the clean long-term path and is more reviewable than continuing to add special-case clamps in hooks.
|
||||
Reference in New Issue
Block a user