Crate switchyard_libsy

Expand description

§switchyard-libsy

Provider-neutral orchestration for multi-LLM optimization. A libsy Algorithm decides which model targets to call, in what order, and how to combine their results. It hands every call back to the host rather than making it, allowing it to embed in proxies, gateways, and agent runtimes without owning an HTTP stack.

§Setup

[dependencies]
async-trait = "0.1"
futures = "0.3"
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
tokio = { version = "1", features = ["macros", "rt"] }

§Built-in algorithms

Type	Purpose
`Passthrough`	Always call one configured target.
`Random`	Select among any number of targets using uniform or weighted routing.
`LlmTaskClassifier`	Ask a judge model to choose an efficient or capable target.
`StageRouter`	Route coding-agent turns from tool and progress signals, with an optional judge fallback.

Noop is a test helper, not a production routing algorithm.

§How it fits together

LlmTarget names a routing destination. An Algorithm selects targets and records Decisions, offloading every model call to its caller: Algorithm::run_stream yields a Step stream whose Step::CallModel items the host serves over its own transport. libsy makes no network calls itself — switchyard-llm-client’s run is a ready-made consumer that drives the stream and performs the calls over HTTP.

The provider-neutral Request, Response, Usage, and LlmResponse contracts come from switchyard-protocol.

§License

Licensed under the Apache License, Version 2.0.

Structs§

AffinityRouter: Retains a model per request identity and forces it on later matching requests.
CallModel: An offloaded model call, surfaced inside Step::CallModel.
ClassifierContractConfig: User-configurable parts of a classifier’s prompt and verdict contract.
CodingAgentDimensions: The two-axis feature view of a single ToolSignals.
CustomClassifierConfig: Settings for a classifier whose JSON Schema and target-selection policy are user supplied.
Driver: How an algorithm’s route makes model calls.
EscalationJudgeConfig: The tuning surface for the trajectory judge.
HandoffNoteConfig: The notes a stage router hands the model it routed to, and the gate deciding which one a turn earns.
LlmFallback: The capability judge a stage router falls through to.
LlmTarget: A named routing target an algorithm routes by. Serving its calls is the stream consumer’s concern: the selected identifier reaches the consumer as decision.selected_model_id() on the offloaded CallModel.
LlmTargetSet: The set of targets an algorithm may route among. An algorithm is constructed with one and picks targets by position (targets) or by name (get_target).
LlmTaskClassifier: Routes requests through a capability, escalation, or custom classifier mode.
Noop: Test helper that returns a hard-coded response without routing or model I/O.
Passthrough: Routing algorithm that always calls one configured target.
Random: Random router implemented as a stateless fall-through composition.
RandomClassifier: Stateless weighted classifier used by random fall-through routing.
Score: One classifier’s recommendation of a routing target, with a [0.0, 1.0] confidence.
ScoreResult: A signed score in (-1, +1) and its magnitude. confidence == score.abs().
StageClassifier: Signal-only stage-router classifier: scores each turn onto the capable/efficient tiers from tool-result signals, via the configured picker mode and the confidence the scorer must reach before it acts on the signal alone.
StageRouter: Routes coding-agent turns between a capable and an efficient tier: tool signals decide first, an optional capability judge takes the turns they cannot, and the picker’s default tier closes the cascade so a turn is never left unrouted.
StageRouterConfig: How a stage router scores turns, and what it hands the model it picks.
StageTargets: The targets a stage router’s two tiers route to.
State: Routing facts accumulated across one session’s algorithm runs.
SubagentOverride: Scores a fixed worker target for delegated sub-agent work; abstains otherwise.
SystemPromptProcessor: Prepends the routed target’s system prompt to the outbound request.
TargetPrompts: System prompts keyed by routing target. A target left unset is routed untouched.
TaskClassifierConfig: Settings that control capability classifier prompting and routing.
ToolSignals: Tool-execution signals extracted from a normalized Request.

Enums§

Classification: A classifier’s verdict for a request: a set of target Scores, flagged by how confident the classifier is that they are decisive.
CustomClassifierPolicy: Policy that maps a custom classifier verdict to a routing target.
DecisionSource: What produced a decision — for stats and explainability.
DriverError: Failures in the step-stream driver.
Event: An event observed by the algorithm. Events are consumed by Processor to mutate state.
LibsyError: Failures surfaced while selecting a route, driving an algorithm, or serving a model call.
LlmClassifierConfig: Complete construction settings for one LLM classifier mode.
PickOutcome: Outcome of pick_tier: either a resolved decision, or a signal that the caller should consult its (impl-specific, async) classifier.
PickerMode: Which tier to default to when the scorer is not confident.
StateValue: A value in a session’s State.
Step: One item in the stream returned by Algorithm::run_stream.
Tier: The two tiers a turn can route to.

Constants§

DECISION_SOURCE_KEY: State.extra key under which the turn’s DecisionSource is recorded.
DEFAULT_RECENT_WINDOW: Default sliding-window size for recent_* counts and windowed severity.

Traits§

Algorithm: An optimization strategy. Implement route; callers drive it with run_stream, serving each Step::CallModel it emits. switchyard-llm-client’s run is the ready-made consumer that does this over HTTP.
Classifier: Scores targets from the current request and the composition’s state.
Processor: Collects events as the algorithm runs and mutates the composition’s state.

Functions§

append_note: Appends note to the request as conversation text.
dimensions_from_signal: Project a ToolSignals onto the two-axis dimension space.
drive: Drive Algorithm::run_stream to completion, handing each offloaded call to serve.
initialize_metrics: Registers process-wide compatibility gauges with the global meter provider.
pick_tier: Decide a turn’s tier from its signal.
score_signal: Score a signal: weighted sum of the dimensions, tanh-squashed.

Type Aliases§

Result: Result type returned by libsy APIs.
StepStream: A boxed, Send stream of Steps — the output of Algorithm::run_stream. Boxed so the trait method that produces it keeps Arc<dyn Algorithm> object-safe.