Crate switchyard_libsy

Expand description

§switchyard-libsy

Provider-neutral orchestration for multi-LLM optimization. A libsy Algorithm decides which model targets to call, in what order, and how to combine their results. It can use target-owned clients or hand each call back to the host, allowing it to embed in proxies, gateways, and agent runtimes without owning an HTTP stack.

§Setup

[dependencies]
async-trait = "0.1"
futures = "0.3"
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
tokio = { version = "1", features = ["macros", "rt"] }

§Quick start

This complete src/main.rs routes a request with LlmTaskClassifier. The first Algorithm::run lets libsy call each target’s client. The second drives Algorithm::run_stream so the host can perform or override each model call itself.

use std::sync::Arc;

use async_trait::async_trait;
use futures::StreamExt;
use switchyard_libsy::{
    Algorithm, LibsyError, LlmClassifierConfig, LlmTarget, LlmTaskClassifier, Step,
    TaskClassifierConfig,
};
use switchyard_protocol::{
    AggLlmResponse, ContentBlock, Context, Decision, LlmClientError, LlmRequest,
    LlmResponse, Message, Request, Response, ResponseOutput, Role, RoutedLlmClient,
};

struct DemoClient;

#[async_trait]
impl RoutedLlmClient for DemoClient {
    async fn call(
        &self,
        _ctx: Context,
        _request: Request,
        decision: Arc<dyn Decision>,
    ) -> Result<Response, LlmClientError> {
        let model = decision.selected_model();
        let text = if model == "judge" {
            r#"{
                "recommended_route": "efficient",
                "p_solve": 0.9,
                "confidence": 0.9,
                "abstain": false,
                "capability_boundary": "supported",
                "primary_rule": "SUP-1",
                "crux": "bounded task"
            }"#
            .to_string()
        } else {
            format!("answer from {model}")
        };
        Ok(Response {
            llm_response: LlmResponse::Agg(AggLlmResponse {
                model: Some(model.to_string()),
                outputs: vec![ResponseOutput {
                    role: Role::Assistant,
                    content: vec![ContentBlock::Text { text }],
                    stop_reason: None,
                }],
                ..AggLlmResponse::default()
            }),
            metadata: None,
        })
    }
}

#[tokio::main(flavor = "current_thread")]
async fn main() -> switchyard_libsy::Result<()> {
    let client = Arc::new(DemoClient);
    let target = |name: &str| LlmTarget {
        semantic_name: name.to_string(),
        llm_client: Some(client.clone()),
    };

    let router: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(
        LlmClassifierConfig::Capability {
            judge_target: target("judge"),
            efficient_target: target("efficient"),
            capable_target: target("capable"),
            config: TaskClassifierConfig {
                base_threshold: 0.5,
                ..TaskClassifierConfig::default()
            },
        },
    )?);
    let request = Request {
        llm_request: LlmRequest {
            model: Some("auto".to_string()),
            messages: vec![Message {
                role: Role::User,
                content: vec![ContentBlock::Text {
                    text: "Explain tail latency".to_string(),
                }],
            }],
            ..LlmRequest::default()
        },
        ..Request::default()
    };

    // `run` serves the judge and routed calls through their target clients.
    let (_, response) = router
        .clone()
        .run(Context::default(), request.clone())
        .await?;
    println!("run returned {}", response.selected_model().unwrap_or("unknown"));

    // `run_stream` exposes those calls so the host controls their transport.
    let stream = router.run_stream(Context::default(), request, None);
    tokio::pin!(stream);
    while let Some(step) = stream.next().await {
        match step? {
            Step::CallLlm(call) => {
                let routed = call.get_routed().clone();
                let target = routed.decision.selected_model().to_string();
                let result = client
                    .call(routed.ctx, routed.request, routed.decision)
                    .await
                    .map_err(|source| LibsyError::client_call(target, source));
                call.respond(result)?;
            }
            Step::Decision(decision) => {
                println!("run_stream chose {}", decision.selected_model());
            }
            Step::ReturnToAgent(response) => {
                println!(
                    "run_stream returned {}",
                    response.selected_model().unwrap_or("unknown")
                );
            }
        }
    }
    Ok(())
}

§Built-in algorithms

Type	Purpose
`Passthrough`	Always call one configured target.
`Random`	Select among any number of targets using uniform or weighted routing.
`LlmTaskClassifier`	Ask a judge model to choose an efficient or capable target.
`StageRouter`	Route coding-agent turns from tool and progress signals, with an optional judge fallback.

Noop is a test helper, not a production routing algorithm.

§How it fits together

LlmTarget pairs a semantic routing name with an optional RoutedLlmClient. An Algorithm selects targets and records Decisions. Use Algorithm::run with target-owned clients, or Algorithm::run_stream when the host owns model transport. The provider-neutral Request, Response, Usage, and LlmResponse contracts come from switchyard-protocol.

§Examples

§License

Licensed under the Apache License, Version 2.0.

Structs§

AffinityRouter: Retains a model per request identity and forces it on later matching requests.
CallLlmRequest: The host-facing half of an offloaded model call, surfaced inside Step::CallLlm.
ClassifierContractConfig: User-configurable parts of a classifier’s prompt and verdict contract.
CodingAgentDimensions: The two-axis feature view of a single ToolSignals.
CustomClassifierConfig: Settings for a classifier whose JSON Schema and target-selection policy are user supplied.
Driver: The offload channel handed to an algorithm’s create_run_task. The algorithm makes model calls with call_llm_target (or call_llm) and publishes its Decisions with info; each call is offloaded to the request’s Step stream and awaits the consumer’s response. The step channel is bounded, so the consumer paces the algorithm one step at a time.
EscalationJudgeConfig: The tuning surface for the trajectory judge.
HandoffNoteConfig: The notes a stage router hands the model it routed to, and the gate deciding which one a turn earns.
LlmCallObservation: One completed model call observed at the algorithm offload boundary.
LlmFallback: The capability judge a stage router falls through to.
LlmTarget: A named routing target: a semantic_name an algorithm routes by, and an optional RoutedLlmClient to serve its calls. An algorithm hands a target to Driver::call_llm_target; the client rides along as RoutedRequest::default_client for the stream consumer to serve or override.
LlmTargetSet: The set of targets an algorithm may route among. An algorithm is constructed with one and picks targets by position (targets) or by name (get_target).
LlmTaskClassifier: Routes requests through a capability, escalation, or custom classifier mode.
Noop: Test helper that returns a hard-coded response without routing or model I/O.
NoopDecision: Test decision carrying the inbound model or a fixed placeholder.
Passthrough: Routing algorithm that always calls one configured target.
PassthroughDecision: Decision emitted before Passthrough calls its configured target.
Random: Random router implemented as a stateless fall-through composition.
RandomClassifier: Stateless weighted classifier used by random fall-through routing.
RoutedRequest: A request paired with the routing Decision that produced it — the offload payload a host reads (via CallLlmRequest::get_routed) to serve the call.
Score: One classifier’s recommendation of a routing target, with a [0.0, 1.0] confidence.
ScoreResult: A signed score in (-1, +1) and its magnitude. confidence == score.abs().
StageClassifier: Signal-only stage-router classifier: scores each turn onto the capable/efficient tiers from tool-result signals, via the configured picker mode and the confidence the scorer must reach before it acts on the signal alone.
StageRouter: Routes coding-agent turns between a capable and an efficient tier: tool signals decide first, an optional capability judge takes the turns they cannot, and the picker’s default tier closes the cascade so a turn is never left unrouted.
StageRouterConfig: How a stage router scores turns, and what it hands the model it picks.
StageTargets: The targets a stage router’s two tiers route to.
State: Routing facts accumulated across one session’s algorithm runs.
SubagentOverride: Scores a fixed worker target for delegated sub-agent work; abstains otherwise.
SystemPromptProcessor: Prepends the routed target’s system prompt to the outbound request.
TargetPrompts: System prompts keyed by routing target. A target left unset is routed untouched.
TaskClassifierConfig: Settings that control capability classifier prompting and routing.
ToolSignals: Tool-execution signals extracted from a normalized Request.

Enums§

Classification: A classifier’s verdict for a request: a set of target Scores, flagged by how confident the classifier is that they are decisive.
CustomClassifierPolicy: Policy that maps a custom classifier verdict to a routing target.
DecisionSource: What produced a decision — for stats and explainability.
DriverError: Failures in the type-erased promise-over-stream driver.
Event: An event observed by the algorithm. Events are consumed by Processor to mutate state.
LibsyError: Failures surfaced while selecting a route, driving an algorithm, or serving a model call.
LlmClassifierConfig: Complete construction settings for one LLM classifier mode.
PickOutcome: Outcome of pick_tier: either a resolved decision, or a signal that the caller should consult its (impl-specific, async) classifier.
PickerMode: Which tier to default to when the scorer is not confident.
RunObservation: One request-scoped observation emitted by the algorithm runner.
StateValue: A value in a session’s State.
Step: One item in the stream returned by Driver::stream / Algorithm::run_stream.
Tier: The two tiers a turn can route to.

Constants§

DECISION_SOURCE_KEY: State.extra key under which the turn’s DecisionSource is recorded.
DEFAULT_RECENT_WINDOW: Default sliding-window size for recent_* counts and windowed severity.

Traits§

Algorithm: An optimization strategy. Implement create_run_task; callers drive it with the provided run (serve calls, get the answer) or run_stream (drive the Step stream yourself).
Classifier: Scores targets from the current request and the composition’s state.
Processor: Collects events as the algorithm runs and mutates the composition’s state.

Functions§

append_note: Appends note to the request as conversation text.
dimensions_from_signal: Project a ToolSignals onto the two-axis dimension space.
initialize_metrics: Registers process-wide compatibility gauges with the global meter provider.
pick_tier: Decide a turn’s tier from its signal.
score_signal: Score a signal: weighted sum of the dimensions, tanh-squashed.

Type Aliases§

RandomDecision: Compatibility name for the decision produced by Random.
Result: Result type returned by libsy APIs.
RunObserver: Request-scoped callback for algorithm-run observations.
StepStream: A boxed, Send stream of Steps — the output of Algorithm::run_stream. Boxed so the trait method that produces it keeps Arc<dyn Algorithm> object-safe.