LLM Classifier Routing¶

LLM classifier routing supports capability classification, trajectory escalation, and custom schema-driven routing across two or more targets.

Configure a classifier route¶

This example uses the packaged classifier prompt as intended: it estimates whether the weak target can complete the task, and keeps the first routing decision for later requests in the same conversation.

schema_version = 1

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[targets.classifier]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[targets.weak]
id = "z-ai/glm-5.2"
llm_client = "openrouter"

[routes.smart]
id = "smart"
type = "llm_classifier"
mode = "capability"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5
threshold_step = 0.1
classify_trigger = "new_session"
message_hash_fallback = true

message_hash_fallback is best-effort: independent sessions with the same first user message share an affinity key. Prefer an explicit x-switchyard-session-id when repeated opening prompts are possible.

The target table names are local references. Their id values are the model identifiers sent to the upstream provider. The route's id, smart, is the model name clients send to Switchyard.

How the decision works¶

The classifier target returns a structured verdict containing:

p_solve: the estimated probability that the weak model completes the task.
capability_boundary: supported, uncertain, unsupported, or unmatched.
primary_rule: the capability-card rule that determines the boundary.
crux: the hardest material requirement for whole-task success.

For a usable verdict, Switchyard routes to weak_target when p_solve is greater than or equal to the applicable threshold. Otherwise it routes to strong_target:

supported uses base_threshold.
uncertain and unmatched use base_threshold + threshold_step.
unsupported uses base_threshold + 2 * threshold_step.

An invalid, inconsistent, or unparseable verdict, or a judge failure, routes to strong_target. Raising either knob sends more traffic to the strong model.

Judge model compatibility¶

The judge must return complete, schema-valid JSON in normal assistant content. Switchyard does not parse provider-specific reasoning fields such as reasoning_content. If content is empty or unparseable, the route falls back to strong_target even when the judge request returned HTTP 200. With session affinity, that fallback can be reused without another judge call.

Capability and escalation routes use JSON Schema structured output by default. For a provider that supports JSON Object mode but not JSON Schema, set response_format_type = "json_object" on the route. Switchyard then adds the verdict schema to the judge prompt and validates the returned object locally.

When a vLLM-compatible provider supports enable_thinking, configure it on the judge target through extra_body:

[targets.classifier]
extra_body = { chat_template_kwargs = { enable_thinking = false } }

enable_thinking is a provider-specific vLLM option, not a general requirement for reasoning models. Other model/provider pairs may work with reasoning enabled or use a different control. Verify the judge response shape before deployment. If reasoning remains enabled, set max_output_tokens high enough for both the reasoning and final JSON. A truncated verdict has the same fail-open result. See the target-level extra_body reference for the server merge behavior.

Tuning options¶

Key	Default	Meaning
`base_threshold`	required	Lowest `p_solve` that routes a supported task to `weak_target`. Must be between `0` and `1`.
`threshold_step`	`0.0`	Amount added for each boundary step. Must be finite and non-negative, and `base_threshold + 2 * threshold_step` must not exceed `1`.
`recent_turn_window`	unset	When unset, the judge sees the opening user task and the latest user message when they differ. When set to `N`, it sees the opening user task and the last `N` conversation messages after that task. `0` keeps only the opening task. Client system and developer instructions are not shown to the judge.
`classify_trigger`	`every_request`	When the judge runs. `every_request` judges every request, tool continuations included. `user_turn` judges each new user message and holds that target across the tool calls between. `new_session` judges once and reuses that target for the session.
`message_hash_fallback`	`false`	When session metadata is absent, keys affinity from the first user-message text. Requires `classify_trigger = "new_session"`.
`prompt`	packaged capability prompt	Replaces the classifier's system prompt. The packaged verdict schema and routing policy remain active.
`response_format_type`	`json_schema`	Structured-output mode for capability and escalation judges. Use `json_object` for providers without JSON Schema support.
`max_output_tokens`	`4096`	Maximum completion tokens available to the classifier verdict. Must be at least `1`.

Override the classifier prompt¶

Set prompt on the route when the packaged capability rubric does not describe your weak model. Do not copy the response schema into the prompt: Switchyard supplies it according to response_format_type.

[routes.smart]
id = "smart"
type = "llm_classifier"
mode = "capability"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5
prompt = """
Estimate whether the weak target can complete the request.
Return exactly one JSON object matching the response schema supplied with the request.
"""

The override changes the instructions only. The judge must still return the packaged crux, primary_rule, capability_boundary, and p_solve fields.

Custom multi-target routing¶

Custom mode accepts an inner JSON Schema and a policy that reads the validated verdict. This example routes across four configured targets:

[routes.smart]
id = "smart"
type = "llm_classifier"
mode = "custom"
classifier_target = "classifier"
targets = ["fast", "balanced", "reasoning", "premium"]
default_target = "premium"
prompt = """
Choose the best configured target for this request.
Return JSON matching the response schema supplied with the request.
"""
response_schema = '''
{
  "type": "object",
  "properties": {
    "decision": {
      "type": "object",
      "properties": {
        "target": {
          "type": "string",
          "enum": ["fast", "balanced", "reasoning", "premium"]
        }
      },
      "required": ["target"],
      "additionalProperties": false
    }
  },
  "required": ["decision"],
  "additionalProperties": false
}
'''

[routes.smart.policy]
type = "target_selector"
selector = "/decision/target"

The names in targets reference existing target tables. Switchyard passes the schema to the provider in a strict structured-output wrapper and validates the returned JSON again. jsonptr resolves the selector against that verdict. A missing, non-string, or unknown target falls back to default_target.

This separation applies to every classifier mode. Prompts containing the legacy {{RESPONSE_SCHEMA}} placeholder are rejected during configuration validation.

Forecast and policy assumptions¶

The packaged prompt forecasts whole-task success for a generic efficient agent. It produces a probability and capability boundary but does not choose a route. The deterministic policy applies base_threshold and threshold_step after generation.

Without affinity, the runtime judges every request. By default, it sends the opening task and the latest user follow-up when they differ. Set recent_turn_window when intervening conversation context affects the forecast. If a client sends only a follow-up fragment without the opening task, enable affinity or include the task history. Threshold tuning changes routing policy; it cannot recover missing task context.

When the judge runs¶

classify_trigger sets how often the target is re-decided.

every_request, the default, judges every request. In an agentic session that includes every tool continuation, so twenty tool steps means twenty-one classifications of one task, and the target can change between any two of them.

user_turn judges each new user message and holds that target through the tool calls that follow:

[routes.smart]
classify_trigger = "user_turn"

Tool results are the agent continuing work the user already asked for, so they do not re-open the decision. A failed or unusable verdict keeps the current target. When no target has been selected yet, the next request is judged again.

new_session judges once and reuses that target for the rest of the session, including strong_target when it was selected as the fallback for an unusable verdict. There is no warmup period, and later requests skip the judge entirely.

The selection is held in per-session state, so requests without a session identity are judged every time. Clients can send x-switchyard-session-id, or enable message_hash_fallback to key on the first user-message text under new_session.

Run the route¶

After installing the Rust server, export the provider credential, validate the configuration, and start the release binary:

export OPENROUTER_API_KEY="your-openrouter-key"  # pragma: allowlist secret
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml \
  --host 127.0.0.1 --port 4000

Send a request using the route ID:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"smart","messages":[{"role":"user","content":"Explain why the sky appears blue."}]}'

Treat the selected target as model-dependent output, not a fixed test result.