Crate switchyard_libsy

Expand description

§switchyard-libsy

Provider-neutral orchestration for multi-LLM optimization. A libsy Algorithm decides which model targets to call, in what order, and how to combine their results. It hands every call back to the host rather than making it, allowing it to embed in proxies, gateways, and agent runtimes without owning an HTTP stack.

§Setup

[dependencies]
async-trait = "0.1"
futures = "0.3"
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
tokio = { version = "1", features = ["macros", "rt"] }

§Built-in algorithms

Type	Purpose
`Passthrough`	Always call one configured target.
`Random`	Select among any number of targets using uniform or weighted routing.
`LlmTaskClassifier`	Ask a judge model to choose an efficient or capable target.
`StageRouter`	Route coding-agent turns from tool and progress signals, with an optional judge fallback.

Noop is a test helper, not a production routing algorithm.

§How it fits together

LlmTarget names a routing destination. An Algorithm selects targets and records Decisions, offloading every model call to its caller: Algorithm::run_stream yields a Step stream whose Step::CallLlm items the host serves over its own transport. libsy makes no network calls itself — switchyard-llm-client’s run is a ready-made consumer that drives the stream and performs the calls over HTTP.

The provider-neutral Request, Response, Usage, and LlmResponse contracts come from switchyard-protocol.

§License

Licensed under the Apache License, Version 2.0.

Structs§

AffinityRouter: Retains a model per request identity and forces it on later matching requests.
CallLlmRequest: The host-facing half of an offloaded model call, surfaced inside Step::CallLlm.
ClassifierContractConfig: User-configurable parts of a classifier’s prompt and verdict contract.
CodingAgentDimensions: The two-axis feature view of a single ToolSignals.
CustomClassifierConfig: Settings for a classifier whose JSON Schema and target-selection policy are user supplied.
Driver: The offload channel handed to an algorithm’s create_run_task. The algorithm makes model calls with call_llm and publishes its Decisions with info; each call is offloaded to the request’s Step stream and awaits the consumer’s response. The step channel is bounded, so the consumer paces the algorithm one step at a time.
EscalationJudgeConfig: The tuning surface for the trajectory judge.
HandoffNoteConfig: The notes a stage router hands the model it routed to, and the gate deciding which one a turn earns.
LlmCallObservation: One completed model call observed at the algorithm offload boundary.
LlmFallback: The capability judge a stage router falls through to.
LlmTarget: A named routing target an algorithm routes by. Serving its calls is the stream consumer’s concern: the target’s name reaches the consumer as decision.selected_model() on the offloaded RoutedRequest.
LlmTargetSet: The set of targets an algorithm may route among. An algorithm is constructed with one and picks targets by position (targets) or by name (get_target).
LlmTaskClassifier: Routes requests through a capability, escalation, or custom classifier mode.
Noop: Test helper that returns a hard-coded response without routing or model I/O.
NoopDecision: Test decision carrying the inbound model or a fixed placeholder.
Passthrough: Routing algorithm that always calls one configured target.
PassthroughDecision: Decision emitted before Passthrough calls its configured target.
Random: Random router implemented as a stateless fall-through composition.
RandomClassifier: Stateless weighted classifier used by random fall-through routing.
RoutedRequest: A request paired with the routing Decision that produced it — the offload payload a host reads (via CallLlmRequest::get_routed) to serve the call.
Score: One classifier’s recommendation of a routing target, with a [0.0, 1.0] confidence.
ScoreResult: A signed score in (-1, +1) and its magnitude. confidence == score.abs().
StageClassifier: Signal-only stage-router classifier: scores each turn onto the capable/efficient tiers from tool-result signals, via the configured picker mode and the confidence the scorer must reach before it acts on the signal alone.
StageRouter: Routes coding-agent turns between a capable and an efficient tier: tool signals decide first, an optional capability judge takes the turns they cannot, and the picker’s default tier closes the cascade so a turn is never left unrouted.
StageRouterConfig: How a stage router scores turns, and what it hands the model it picks.
StageTargets: The targets a stage router’s two tiers route to.
State: Routing facts accumulated across one session’s algorithm runs.
SubagentOverride: Scores a fixed worker target for delegated sub-agent work; abstains otherwise.
SystemPromptProcessor: Prepends the routed target’s system prompt to the outbound request.
TargetPrompts: System prompts keyed by routing target. A target left unset is routed untouched.
TaskClassifierConfig: Settings that control capability classifier prompting and routing.
ToolSignals: Tool-execution signals extracted from a normalized Request.

Enums§

Classification: A classifier’s verdict for a request: a set of target Scores, flagged by how confident the classifier is that they are decisive.
CustomClassifierPolicy: Policy that maps a custom classifier verdict to a routing target.
DecisionSource: What produced a decision — for stats and explainability.
DriverError: Failures in the type-erased promise-over-stream driver.
Event: An event observed by the algorithm. Events are consumed by Processor to mutate state.
LibsyError: Failures surfaced while selecting a route, driving an algorithm, or serving a model call.
LlmClassifierConfig: Complete construction settings for one LLM classifier mode.
PickOutcome: Outcome of pick_tier: either a resolved decision, or a signal that the caller should consult its (impl-specific, async) classifier.
PickerMode: Which tier to default to when the scorer is not confident.
RunObservation: One request-scoped observation emitted by the algorithm runner.
StateValue: A value in a session’s State.
Step: One item in the stream returned by Driver::stream / Algorithm::run_stream.
Tier: The two tiers a turn can route to.

Constants§

DECISION_SOURCE_KEY: State.extra key under which the turn’s DecisionSource is recorded.
DEFAULT_RECENT_WINDOW: Default sliding-window size for recent_* counts and windowed severity.

Traits§

Algorithm: An optimization strategy. Implement create_run_task; callers drive it with run_stream, serving each Step::CallLlm it emits. switchyard-llm-client’s run is the ready-made consumer that does this over HTTP.
Classifier: Scores targets from the current request and the composition’s state.
Processor: Collects events as the algorithm runs and mutates the composition’s state.

Functions§

algorithm_label: The algorithm label Algorithm::run_stream stamps into a request context; empty until stamped.
append_note: Appends note to the request as conversation text.
dimensions_from_signal: Project a ToolSignals onto the two-axis dimension space.
drive: Drive Algorithm::run_stream to completion, handing each offloaded call to serve.
initialize_metrics: Registers process-wide compatibility gauges with the global meter provider.
pick_tier: Decide a turn’s tier from its signal.
score_signal: Score a signal: weighted sum of the dimensions, tanh-squashed.

Type Aliases§

RandomDecision: Compatibility name for the decision produced by Random.
Result: Result type returned by libsy APIs.
RunObserver: Request-scoped callback for algorithm-run observations.
StepStream: A boxed, Send stream of Steps — the output of Algorithm::run_stream. Boxed so the trait method that produces it keeps Arc<dyn Algorithm> object-safe.