Stage Router¶
Execution routing uses the stage router to choose a model from tool results
and agent progress. Configure it with type = "stage_router".
Stage-router routing sends each request to either a capable model or a
typically cheaper efficient one, depending on where the agent is in its run.
The goal is to spend the capable model on turns that need exploration or error
recovery and let the efficient model carry routine, mechanical work. Choose a
required picker for ambiguous turns, tune the split with
confidence_threshold, and optionally add an LLM classifier.
Model-call fallback happens after the routing decision. Eligible context-window or availability failures can try the route's other completion target. See Context-Window Handling.
How it works¶
A tool-using agent's run moves through stages that call for different amounts of model capability. The built-in vocabulary is calibrated for coding agents: it recognizes file observation, mutation, planning, shell activity, and test results.
For each LLM call without an active hold, stage-router estimates the agent's state from the conversation's tool-result history, scoring signals in two directions:
- RECOVERY → capable:
severity(windowed error severity),spinning(deep churn with no reads or writes), andexploring(reading or planning without producing) push toward the capable tier. - PROGRESS → efficient:
production_intensity(the share of recent recognized operations that are writes or edits) pushes toward the efficient tier with 1.5 times the weight of one signal. Production with no error in the recent window adds 1.2 more signals toward efficient, so clean production resolves a turn to the efficient tier on its own, even when writes are a small share of recent operations.
The axes are corroborative: the signed score is tanh-squashed to a
confidence in [0, 1]. One maxed scoring dimension produces about 0.46;
corroborating evidence pushes confidence decisively past a 0.5 threshold.
Repeated failures, critical-error severity, and context compaction are hard
overrides to the capable tier. Set compaction_override = false to disable the
compaction override in either picker. Active holds bypass signal extraction and scoring.
Severity and test results come from tool output that ran something. The contents
returned by the built-in read and search tools do not count, unless the tool
reports a failure. Reading a missing file does not add error severity, even
when the tool marks the result as an error.
confidence_threshold sets how sure that estimate must be before the router acts
on the signal alone. Scores inside the ambiguous band go to the optional
classifier and then, if it cannot decide, to the picker's default tier. A turn
with no tool-result history follows the same fallback path.
The routing decision for one turn:
%%{init: {"flowchart": {"curve": "linear", "nodeSpacing": 36, "rankSpacing": 38}}}%%
flowchart TB
start(["New turn"])
hold{"Active hold?"}
heldRoute(["Route to held tier<br/>confidence = 1.0"])
history[/"Read tool-result history"/]
recovery{"Hard recovery signal?"}
scorer["Compute signed tool-signal score"]
band{"Outside the<br/>ambiguous band?"}
classifier{"Usable classifier verdict?"}
capable(["Route to CAPABLE"])
signalRoute(["Route by score sign<br/>negative → efficient · positive → capable"])
classifierRoute(["Route by classifier verdict"])
defaultRoute(["Route to picker default"])
start --> hold
hold -->|Yes| heldRoute
hold -->|No| history
history --> recovery
recovery -->|Yes| capable
recovery -->|No| scorer
scorer --> band
band -->|Yes| signalRoute
band -->|No · ambiguous| classifier
classifier -->|Yes| classifierRoute
classifier -->|No · absent or invalid| defaultRoute
classDef input fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e,stroke-width:2px;
classDef process fill:#f8fafc,stroke:#64748b,color:#1e293b,stroke-width:1.5px;
classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px;
classDef capable fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:3px;
classDef outcome fill:#dcfce7,stroke:#16a34a,color:#14532d,stroke-width:2px;
class start,history input;
class scorer process;
class hold,recovery,band,classifier decision;
class capable capable;
class heldRoute,signalRoute,classifierRoute,defaultRoute outcome;
Both pickers support capable_hold_turns (default 2) and
efficient_hold_turns (default 0 on standalone routes; composite's
capable-first policy defaults to -1). A positive value holds that tier for the
specified number of requests after a switch, excluding the switch request.
Held requests do not restart the counter. A finite capable hold also starts or
restarts on a signal-driven capable decision. Active holds return the selected
tier at confidence 1.0 before extracting tool signals or scoring. The existing
cascade then skips the fallback judge. Passing tests and errors do not end holds
early, and held requests receive no handoff notes.
Set either value to 0 to disable that hold, or -1 to pin to that tier after
switching into it. The initial selection does not activate a pin. Pins override
later signals, including passing tests and critical errors, and skip the fallback
judge. If both values are -1, the first tier switch pins its destination.
Switches chosen by signals, the fallback judge, or the picker default all count.
Holds use the existing session state and are separate for each agent ID. Supply
a stable session ID across requests. A pin ends when that session state is cleared
(including session completion, expiry, or a server restart). Values below -1 or
above 4294967295 are rejected.
Pickers¶
The picker name says which tier is the default: the tier used when the signals are ambiguous and no classifier verdict is available.
efficient_first: efficient is the default; escalate to capable only when the signals (or the classifier) clearly say so. Cost-first.capable_first(experimental): capable is the default; drop to efficient only when the signals (or the classifier) clearly say so. Quality-first.
Both pickers read the same signals; only the default tier differs.
capable_first is experimental
Every published threshold and routing result comes from efficient_first
runs. capable_first works and the server accepts it, but it has not been
benchmarked, so there are no calibrated thresholds for it and no measured
accuracy or cost figures to set expectations against. The server logs a
warning at startup when a route selects it. Use efficient_first unless you
are running your own calibration. See
Reachable scores per picker for what a
threshold can and cannot do with each picker.
Tuning confidence_threshold¶
The tool-signal scorer gives each turn a signed score in (-1, 1): negative
scores point to the efficient tier, positive scores point to the capable tier,
and the absolute value is the confidence. confidence_threshold creates a
closed ambiguous band from -threshold to +threshold. Scores outside the
band make a signal-based decision; scores inside it go to the optional
classifier or fall back to the picker's default tier.
The TOML schema requires you to choose picker explicitly; there is no implicit
default.
Set 0.5 explicitly. confidence_threshold is required by the TOML schema;
0.5 is the recommended starting point, derived from many coding benchmarks,
and what the example below uses.
Reachable scores¶
The scorer adds a few signals from the recent tool results. Error severity and stalls count toward the capable tier. Writes count toward the efficient tier. Each signal is bounded, so a turn's confidence can only reach a few values. These are the ones that matter when you choose a threshold:
| Situation in the recent window | Confidence | Direction |
|---|---|---|
| A soft error alone | 0.21 | Capable |
| A hard error alone, or a stall alone | 0.46 | Capable |
| Every operation is a write, with a soft error | 0.49 | Efficient |
| One write among three operations and no error, the smallest clean-production score at the default window | 0.69 | Efficient |
| A hard error plus a stall | 0.76 | Capable. The most the signals reach in this direction (0.7616). |
| Every operation is a write and no error | 0.87 | Efficient. The most the signals reach in this direction (0.8741). |
Production turns the stall signals off, so with any production in the window
the capable side stays below 0.46. An error in the window removes the
clean-production bonus, so with a hard error the efficient side reaches at most
0.24. The unit test scorer_ceilings_match_the_documented_values in
crates/libsy/src/algorithms/util/stage.rs
pins these values, so a change to the scorer fails CI until this table is
updated.
Reachable scores per picker¶
Both pickers use the same score. The picker only sets which direction the signals must prove.
| Picker | The signals move a turn to | Reachable confidence | What a threshold means |
|---|---|---|---|
efficient_first |
Capable (escalation) | Up to 0.7616 | At 0.5, a hard error alone (0.46) stays in the band, so an escalation needs a second signal, a hard override, or the classifier. Below 0.46, a hard error alone escalates. At 0.77 and above, the signals never escalate; only overrides, capable holds, and the classifier can. |
capable_first |
Efficient (offload) | Up to 0.8741, and at most 0.49 when an error is in the window | At 0.5, clean production offloads (0.69 and up at the default window) and nothing else does. Below 0.49, production with a soft error in the window also offloads. At 0.88 and above, the signals never offload. |
Lower the threshold to let the signals decide more turns. Raise it to keep more
turns on the picker's default tier. Hard overrides, capable holds, and the
classifier do not use the threshold. The server warns at startup that
capable_first has no calibrated threshold.
What 0.5 means with efficient_first¶
With picker = "efficient_first", no classifier, and a threshold of 0.5:
- scores below
-0.5route to efficient from the tool signals; - scores from
-0.5through+0.5are ambiguous and fall back to efficient; - scores above
+0.5route to capable from the tool signals.
Hard overrides and capable-hold state can still select capable independently of this score. Lowering the threshold narrows the ambiguous band, so more turns are decided directly by the scorer. Raising it widens the band, so more turns use the picker default (or the classifier, when configured). That movement changes the efficient/capable routing split.
confidence_threshold |
Behavior | Typical use |
|---|---|---|
0.0 |
Every non-zero score makes a signal-based decision. A neutral score or missing tool history still falls through. | Maximize decisions by the tool-signal scorer. |
0.5 |
Scores from -0.5 through +0.5 are ambiguous. |
Recommended starting point, derived from many coding benchmarks. |
1.0 |
Every ordinary score is ambiguous; hard overrides and capable holds still apply. | Classifier-driven when a classifier is configured; picker-driven otherwise. |
A route with a 1.0 threshold remains valid without a classifier. In that case,
ordinary sub-threshold turns fall back to the picker's default tier; hard
overrides and capable-hold behavior still apply.
The signal-vs-classifier split is dataset-dependent. Measure it in production:
/v1/stats reports decision counts by source and selected target, plus aggregate
scorer metrics. Response headers and structured decision logs explain individual
selections.
Calibrating the threshold from run data¶
Use about 10% of your representative tasks. Replay their agent histories through the Stage tool-signal scorer and collect the unthresholded signed score for each ordinary turn. Track hard overrides and capable holds separately because changing the threshold does not affect them. Plot the score distribution, then overlay candidate ambiguous bands.
For efficient_first, count how many turns fall below the band, inside it, and
above it. Without a classifier, those regions map to signal-selected efficient,
default-efficient, and signal-selected capable decisions. Choose the threshold
that gives the routing split you want, then validate it on the same sample before
running the full task set.
Route configuration¶
Knob reference¶
Every key a stage_router route accepts. The handoff_notes and classifier
tables are optional. The
TOML schema has the full table with
required columns.
| Key | Default | Meaning |
|---|---|---|
capable_target |
required | Capable tier. |
efficient_target |
required | Efficient tier. |
picker |
required | efficient_first or capable_first (experimental). Tier used when the signals are not confident. |
confidence_threshold |
required | Corroboration a decisive pick needs, 0 to 1. Alias efficient_first_threshold. |
capable_first_threshold |
0.0 |
Used only after an outer composite classifier selects the capable tier. No effect on a standalone route. |
capable_system_prompt |
unset | System prompt prepended for capable requests. |
efficient_system_prompt |
unset | System prompt prepended for efficient requests. |
recent_turn_window |
3 |
Trailing tool results the signals are computed over. |
compaction_override |
true |
Context compaction forces capable. Set false to use the other signals and picker normally. |
capable_hold_turns |
2 |
Requests kept on capable after switching to it; signal-driven capable decisions also start finite holds. 0 disables; -1 pins after a switch. |
efficient_hold_turns |
0 |
Requests kept on efficient after switching to it. 0 disables; -1 pins after a switch. |
tool_semantics.observe |
[] |
Domain tool names that count as investigation. |
tool_semantics.mutate |
[] |
Domain tool names that count as production. |
tool_semantics.plan |
[] |
Domain tool names that count as planning. |
tool_semantics.new |
[] |
Domain tool names that count as neutral forward activity. |
handoff_notes.escalation_note |
required in the table | Note handed to the capable tier when the signals or a hard override escalate. |
handoff_notes.deescalation_note |
unset | Note handed to the efficient tier whenever the signals select it, with or without a capable turn before. |
handoff_notes.only_on_wrong_signal_escalation |
true |
When true, restricts escalation notes to signal-driven capable decisions. Held turns receive no notes with either setting. |
classifier.target |
required in the table | Judge target. Not a routing destination. |
classifier.base_threshold |
required for LLM judges | Lowest p_solve that still routes efficient. |
classifier.threshold_step |
0.0 |
Added once for uncertain or unmatched verdicts, twice for unsupported. |
classifier.fail_open |
true |
On judge failure, fall open to the picker default instead of failing the request. |
classifier.decision |
unset | Use a decision model instead of an LLM judge. Excludes the LLM judge keys below. |
classifier.classify_trigger |
every_request |
Only the default is accepted. The judge runs on every ambiguous turn. |
classifier.message_hash_fallback |
false |
Rejected at load. The judge retains no verdict. |
classifier.recent_turn_window |
unset | Trailing messages the judge sees after the opening task. A different unit from the route's recent_turn_window, which counts tool results. |
classifier.prompt |
packaged | Replaces the judge prompt. |
classifier.response_format_type |
json_schema |
json_object for providers without JSON Schema. |
classifier.max_output_tokens |
4096 |
Token budget for the verdict. |
classifier.judge_deadline_ms |
unset | Whole-consultation bound in milliseconds on the judge call, response drain included. On expiry the judge counts as failed and classifier.fail_open applies. Must be at least 1. |
subagents |
unset | Separate policy for delegated sub-agent work. |
classifier.response_schema, classifier.policy, and any classifier.mode
other than capability belong to
composite routes and are rejected
here. classifier.mode = "capability" is accepted and changes nothing.
Minimal route¶
schema_version = 1
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"
[targets.capable]
id = "openai/gpt-4o"
llm_client = "openrouter"
# system_prompt = "diagnose before you edit" # optional
[targets.efficient]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"
# system_prompt = "follow the settled plan" # optional
[routes.stage]
id = "switchyard/stage"
type = "stage_router"
capable_target = "capable"
efficient_target = "efficient"
picker = "efficient_first"
confidence_threshold = 0.5
recent_turn_window = 3 # optional, defaults to 3
capable_hold_turns = 2 # optional, defaults to 2
efficient_hold_turns = 0 # optional; -1 pins after switching to efficient
Save as routes.toml and start the server:
This is the recommended default: routing on tool signals alone, no classifier.
Plan and execute¶
To start with the capable model and keep using the efficient model after a switch, add this route to the minimal configuration:
[routes.plan_execute]
id = "switchyard/plan-execute"
type = "stage_router"
capable_target = "capable"
efficient_target = "efficient"
picker = "capable_first"
confidence_threshold = 0.0
capable_hold_turns = 0
efficient_hold_turns = -1
capable_system_prompt = "Inspect the task and plan before modifying files."
With no tool history, Stage selects capable and adds the planning prompt. After successful writes, Stage can select efficient; critical failures can keep capable selected. A switch from capable to efficient starts a hold that keeps selecting efficient while the session state remains. Later tool errors do not end it. Only capable requests receive the planning prompt. See How it works for scoring and hold behavior.
Send requests to switchyard/plan-execute with the same
x-switchyard-session-id from the first planning request onward. Include the
full conversation, including the plan, tool calls, and results. For the Responses
API, send that history in input and omit both previous_response_id and
conversation. Known continuation IDs bypass Stage even with full input
history. Send x-switchyard-session-final: true when the shared session ends to
clear all of its agents' state.
When migrating from the removed plan_execute type, use this recipe and replace
planning_prompt with capable_system_prompt. Stage has no exact equivalent
for the old judged handoff or reasoning-to-text conversion; keep the plan in
ordinary conversation text so the executor receives it.
Optional: custom tool semantics¶
Extend the built-in coding vocabulary when an agent uses domain-specific tool names. Mappings are route-scoped, additive, and matched by exact name without regard to ASCII case:
[routes.stage.tool_semantics]
observe = ["KB_search", "get_customer_by_phone"]
mutate = ["send_payment_request", "update_inventory"]
plan = ["create_research_plan"]
new = ["start_conversation", "send_message_to_user"]
The categories affect existing stage signals:
observecounts as investigation, like the built-in read and search tools.mutatecounts as production, like the built-in write and edit tools.plancounts as investigation, like the built-in planning tools.newrecords forward activity that suppresses falsespinningandexploringsignals, but does not otherwise favor either tier.- Unmatched tools remain
unknown; there is nounknownconfiguration key.
Configuration cannot reclassify a built-in tool. Empty names, duplicate names across categories, and unknown category keys are rejected when the route is loaded. Argument-aware wrapper tools, inferred semantics, and learned routing rules are outside this exact-name configuration.
MCP tools match by their bare tool name or by their full name. Claude Code sends
an MCP tool as mcp__<server>__<tool>, and Codex sends it with a separate
namespace. For both, send_payment_request matches the send_payment_request
tool on the billing MCP server, and so does
mcp__billing__send_payment_request. A server name that contains __ needs the
full name.
Optional: handoff notes¶
Add a [routes.stage.handoff_notes] section to append contextual guidance to a
request when tool signals decide its tier. Notes exist only in the forwarded
request; they do not accumulate in the caller's conversation. The escalation
note accompanies a signal-driven capable decision, and the optional de-escalation
note accompanies a signal-driven efficient decision.
[routes.stage.handoff_notes]
escalation_note = "the previous model was stalling; pick up the diagnosis"
# deescalation_note = "..." # optional
# only_on_wrong_signal_escalation = true # default; held turns never receive notes
Classifier and picker-default decisions do not add handoff notes.
Optional: LLM classifier fallback¶
The block is optional. Without it, an ambiguous turn goes directly to the
picker tier.
To judge ambiguous turns with a model call, add the block. The classifier runs when the scorer has no tool history or produces a score inside the closed ambiguous band:
[targets.stage_judge]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"
[routes.stage.classifier]
target = "stage_judge" # judge-only; never a completion destination
base_threshold = 0.5 # p_solve floor to route efficient; below this → capable
threshold_step = 0.1 # +0.1 for uncertain/unmatched; +0.2 for unsupported
recent_turn_window = 3 # trailing turns added to the opening task
prompt = "Estimate whether the efficient target can complete this request."
response_format_type = "json_object" # optional; default is "json_schema"
prompt replaces the packaged capability-classifier prompt. In the default
json_schema mode, Switchyard sends the verdict schema through the
structured-output request. Set response_format_type = "json_object" for
providers that only support JSON Object mode; Switchyard then adds the schema to
the judge prompt and validates the returned object locally. The verdict schema
and routing thresholds remain unchanged.
The nested stage table in Composite Routing does not
accept a classifier block. Its outer classifier supplies the tier used when
the stage signals are ambiguous.
Give the classifier its own LLM client or quota bucket where possible. Sharing one provider bucket with the efficient tier adds a request per classified turn and can cause sustained 429s at scale.
Observability¶
When a model serves the request, the response identifies it with this routing header:
| Header | Content |
|---|---|
x-model-router-selected-model |
The model ID that served the response, including any model-call fallback. |
Decision sources¶
The router labels each resolved turn with a decision_source. /v1/stats
counts turns by it under algorithm_stats.stage_router.routing_decisions, and
the stage router override log line carries it for overrides:
| Source | When |
|---|---|
override |
A repeated failure, critical-error severity, or context-compaction marker forced the capable tier. Structured logs set override_reason to repeated_failure, critical_error, or compaction. |
capable_hold |
A hold or pin kept this turn on capable. Any switch to capable can start it; signal-driven capable decisions also start finite holds. |
efficient_hold |
A configured hold or pin kept this turn on the efficient tier. |
dimensions |
The signed score fell outside the ambiguous band and selected a tier. |
ambiguous |
The signed score stayed inside the band. This label never reaches /v1/stats or the logs: the turn continues to the classifier or the picker default, and that later source is recorded. |
llm-classifier |
The signals were ambiguous and the classifier returned a usable verdict. |
fall_open |
The signal scorer abstained and the classifier was absent or could not decide, so the picker default was used. |
When not to use stage-router¶
- Single-model deployments. Use a
passthroughroute instead. - Probabilistic A/B splits. Use
Random Routing (
type = "random"). The stage-router's signals are wasted on a fixed traffic ratio. - No tool-result history. Stage-router needs meaningful tool-call traffic to populate its signals. For pure chat-completion workloads, the signal scorer abstains; the optional classifier decides, or the request uses the picker default when no classifier is configured.
Related¶
- Model Selection: choose and check the two models before tuning the threshold.
- Mechanics of Libsy: the execution model and host responsibilities.
- TOML Schema: the complete configuration reference.
- Tuning: check your current route and compare candidate settings on your workload.