Skip to content

Switchyard v0.3.0 Release Notes

Switchyard 0.3.0 focuses on integrating routing into gateways and agent harnesses. It builds on the Rust server and library introduced in 0.2.0, adds new ways to combine routing policies, and addresses protocol and tool-loop failures found during release testing.

Highlights

Integrate without adopting a separate server

  • NeMo Relay native plugin. Load a Switchyard TOML deployment into Relay and execute routes in process. The plugin shares the server's runner and configuration, translates the three supported API formats, and reports routing decisions, serving-model fallback, usage, and overhead through Relay. See the integration guide.
  • Reusable runner and embedding API. switchyard-runner loads deployments from a file or TOML text. Rust and Python hosts can instead drive libsy's routing steps while retaining control of model calls, credentials, and retries. Successful outcomes include ordered model candidates and optional decision evidence. See the library guide.
  • Decision-only and hosted-server options. The standalone server adds /v1/decision, optional forwarding for unmatched requests, and a Dockerfile. Rust hosts can mount only its inference endpoints and own authentication, model discovery, and operational endpoints separately.
  • LiteLLM routing plugin. The experimental example now runs at LiteLLM's routing-plugin boundary, with Stage and Random configurations. LiteLLM owns provider calls and retries. See the example.

Choose routing behavior for the workload

  • Auto provides the recommended Stage Router preset. It starts on the efficient model and uses execution signals; it does not compare routing strategies at runtime.
  • Composite combines an LLM classifier with Stage Router. The classifier sets the tier used when execution signals are inconclusive. This is one supported pairing, not an arbitrary routing graph.
  • Advisor Gate keeps an executor model on the task while a stronger advisor reviews selected turns. It can approve a turn or request a redo, with review budgets and stall checks.
  • Sub-agent-aware routing applies a separate policy to delegated work. Codex compaction stays with the parent; spawned threads use the child route. Claude Code child-identity headers require version 2.1.139 or newer.
  • More deployment control includes per-target system prompts and reasoning effort, custom Stage Router tool semantics, and explicit classifier triggers for each request, user turn, or session.

Use the routing overview to choose a strategy. Existing Task, Execution, Escalation, and Random routing remain available; they are not all new algorithms in this release.

Keep agent requests and tool loops intact

The release includes fixes across OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages:

  • Preserve supported tool history, parallel-call restrictions, tool allowlists, structured output, reasoning controls and signatures, and URL citations.
  • Preserve supported image, audio, document, and tool-result content during re-encoding, including inline PDFs and URL-based media.
  • Correct streaming tool names, IDs, stop reasons, empty tool inputs, and output delivered only in a terminal Responses snapshot. Unsupported native output fails explicitly when it cannot be translated.
  • Keep stored Responses continuations on the owning target. For Chat or Anthropic targets, reconstruct continuations from in-memory transcripts.
  • Handle context overflow and explicit tool failures in routing, keep zero-weight models out of Random fallback, and reject ambiguous same-model client configurations.

These changes do not make every provider-specific feature interchangeable. Test the fields, media types, tool loop, and state behavior your integration uses. See Responses state and retention.

Diagnose failures without exposing provider credentials

Routing outcomes carry an ID and bounded evidence, with matching Rust, Python, and OpenTelemetry fields. Durable logs add route and caller-origin attribution. Stream metrics reflect the final outcome, fallback metrics identify each candidate, and translated usage retains cache-write and thinking-token counts. Missing usage remains unknown rather than being invented as zero.

Configured provider keys are redacted from server and Relay outputs. Caller-supplied internal translation metadata is stripped. Optional per-client deadlines cover retries and stream reads; exhausted HTTP-client failures stop routing rather than allowing an algorithm to silently select another model.

Libsy does not install an exporter or transmit telemetry itself. The host owns exporters, filtering, and retention. See the OpenTelemetry reference and client timeout contract.

Reproduce evaluations with explicit settings

The release adds a DeepSWE v1.1 Harbor/Pier workflow, Stage Router and Advisor Gate profiles, and qualification settings. The qualification document records the 113-task denominator, Codex and Harbor versions, budgets, retries, scoring, pricing snapshot, and run revisions. Public profiles use a different provider from the qualification runs, so absolute results can differ.

A NeMo Gym tutorial compares fixed-model and routed execution through LiteLLM. Release-soak and routing-overhead tools support operational checks. These workflows do not establish a general cost or accuracy improvement across workloads.

Upgrade from v0.2.0

This is a breaking pre-1.0 upgrade. Pin the source revision or released package version tested by your integration.

If you use Required action
switchyard serve, switchyard launch, or the Python agent wrappers Use the standalone switchyard-server and connect clients directly. The Python CLI and legacy FastAPI/YAML server stack are removed.
Legacy Python chains, profiles, or switchyard.lib imports Migrate to switchyard.libsy and the native TOML/server path. The core package now requires Python 3.10+ and declares no Python runtime dependencies. Install your host's model-client dependencies separately.
Old package extras Remove [server], [cli], [all], [tracing], and [affinity-redis] from dependency declarations. The package no longer defines extras.
Custom Rust or Python routing drivers Use Step::CallModel / Step::Done, supply runtime category-to-model mappings, and consume RoutingOutcome.selected_model_ids. Update source-level types and struct literals against the current API.
session_affinity classifier configuration Replace it with classify_trigger; new_session covers session-level reuse. Supply a stable session ID when session behavior is required.
Metrics dashboards Check outcome labels (ok / error) and removed routing_tier labels. Do not equate HTTP attempts, answer candidates, logical calls, and user requests.
Relay caller-credential forwarding Use deployment-owned credentials. The native plugin rejects forward_auth = true; standalone forwarding has credential-family restrictions.

Start with Getting Started, the TOML schema, and the runnable embedding example. The 0.2.0 packages do not provide the new API. Relay plugin bundles are distributed separately and must be checked against their source-commit and host-compatibility metadata.

Maturity and limitations

  • Component maturity is unchanged: libsy is Beta; the LLM client and runner are Alpha; the standalone server is for demos and evaluation, not production. Server release validation covers Ubuntu 24.04, Linux x86_64. Other platforms are outside that validation scope.
  • Relay: supported versions are >=0.8.0, <1.0.0, but 0.8.x and 0.9.0 have a documented upstream-error propagation issue with native plugins enabled. Follow the compatibility note, including isolating unmanaged traffic in a plugin-disabled gateway.
  • State and privacy: cross-format Responses continuations retain transcripts in process memory. State is not durable across restarts or shared across replicas. Review retention and replica routing before using stored sessions.
  • Credential redaction: replacement operates on individual events. It does not reconstruct secrets split across events or cover arbitrary encodings. This is not a general-purpose data-loss-prevention guarantee.
  • Experimental work: Prefill Router requires user-supplied compatible checkpoints and encoder assets; no supported exporter or checkpoint is provided. The LiteLLM example is experimental. The separate CRAFT task-generation tooling is also experimental, not a production routing feature.

Further reading