Mechanics of Libsy

Libsy expresses work as a stream of Steps. Your application executes that work with its own clients. Done is the exception: it carries the completed routing outcome.

Follow one request through that exchange below. The snippets show parts of an integration; the Rust quickstart has a complete program, imports, and matching development-API dependencies.

Why execution stays with your application

Libsy separates routing policy from model execution. The algorithm decides which models to use and how their results affect the next step. Your application decides how those models are called.

flowchart TD
    route["Libsy algorithm<br/>Choose the next Step"]
    execute["Your application<br/>Run your model or decision client"]
    done["Your application<br/>Use the routing outcome"]

    route -->|"CallModel or CallDecision"| execute
    execute -->|"call.respond(...)"| route
    route -->|"Done"| done

    classDef algorithm fill:#76b9001a,stroke:#76b900,stroke-width:2px
    classDef application fill:transparent,stroke:#6b7280
    class route algorithm
    class execute,done application

A run can finish with Done immediately or request several calls before finishing.

A gateway can fulfill these calls through its existing dispatch, credentials, retries, and observability. That includes calls to judges used during routing. An agent harness can supply its conversation and tool history, then use its model clients to execute the requested work. Both adapt their inputs and outputs to the shared protocol types.

Routing can require more than choosing a model ID. An algorithm may change a request, ask a judge, inspect an answer, or compare results from several models. Returning responses through call.respond lets it continue that work. When a selection alone is enough, Done leaves the final model call to your application.

The shared protocol crate lets clients and algorithms use the same request and response contracts. A client can implement those contracts without depending on the routing algorithm implementation.

Start with a request

Your application supplies a Request. Its llm_request field holds the LlmRequest: the conversation, instructions, tools, and generation settings. These provider-neutral types come from switchyard-protocol.

let request = Request {
    llm_request: LlmRequest {
        messages: vec![Message::text(Role::User, "Fix the failing tests.")],
        ..Default::default()
    },
    metadata: Some(Metadata {
        session_id: Some("session-42".into()),
        ..Default::default()
    }),
    ..Default::default()
};

Metadata holds the session ID above, which algorithms can use to retain session decisions. It can also include a correlation ID for your logs. The optional raw_request field holds data your application retains; libsy does not read it.

Next, supply the models the algorithm can use. RuntimeModels groups model IDs by Category, such as Efficient, Capable, or Judge:

let models = Arc::new(RuntimeModels::new(HashMap::from([
    (Category::Efficient, vec!["Efficient".into()]),
    (Category::Capable, vec!["Capable".into()]),
    (Category::Judge, vec!["Judge".into()]),
    (Category::Any, vec!["Efficient".into(), "Capable".into()]),
])));

These are local example IDs. Within a category, models are ordered best-first. A ModelId names a destination; your application decides which client serves it.

With an algorithm, request, model pool, and initial State, start a run:

let state = State::default();
let mut steps = algorithm.run_stream(request, models, state);

run_stream returns a StepStream. Each item is a Result<Step>: either the next step or an error that ends the run.

Read the next Step

A Step describes one unit of work for your application, or reports that routing has finished. Its three variants are:

pub enum Step {
    CallModel(Box<CallModel>),
    CallDecision(Box<CallDecision>),
    Done(Box<RoutingOutcome>),
}

When you write Step::CallModel(call) in a match, call is the Box<CallModel> stored in that variant. It is an object describing a requested operation, with a way to return its result to the waiting algorithm.

Reuse steps from above to read those operations. Here, client and decision_client are your async adapters; the next sections explain their inputs and results:

while let Some(step) = steps.next().await {
    match step? {
        Step::CallModel(call) => {
            let request = call.request.clone();
            call.respond(client.call(request)).await?;
        }
        Step::CallDecision(call) => {
            let request = call.request.clone();
            call.respond(decision_client.call(request)).await?;
        }
        Step::Done(outcome) => {
            // Inspect the result; final-answer handling is shown below.
            println!("Selected: {}", outcome.selected_model_id()?);
            break;
        }
    }
}

This loop serves one call at a time. Concurrent execution keeps reading steps while calls are running.

For example, a signal-only router can finish locally. A classifier might first ask your client to call a judge:

Signal-only routing:  Done(outcome)
Judge-based routing:  CallModel(call) -> your response -> Done(outcome)
Decision routing:     CallDecision(call) -> your typed response -> Done(outcome)

Step::CallModel

Suppose the next step is CallModel. The CallModel object describes the operation:

// The model has already been set to the first candidate.
let request = call.request.clone();
let first_candidate = &call.models[0];
assert_eq!(request.model_id().as_ref(), Some(first_candidate));

models is a nonempty candidate list in fallback order. Your application executes the attempts and must update the request's model before trying a later candidate. call.algorithm names the requesting algorithm for your own telemetry.

In the Step loop, client.call is your async adapter, returning switchyard_libsy::Result<Response>. It owns the provider call and any format conversion, as in the OpenRouter adapter. call.respond consumes the call and runs that work future, returning its result to the algorithm.

The returned Response contains an LlmResponse in its llm_response field:

pub enum LlmResponse {
    Stream(LlmResponseStream), // Live response events.
    Agg(AggLlmResponse),       // A complete, buffered answer.
}

The LlmResponseStreamEvent provides access to provider-neutral text, tool-call updates, and usage information.

StepStream carries work and the routing outcome. LlmResponse::Stream carries model output. Your client can return that live stream without reading it to completion first.

If you want a shared client interface, switchyard-protocol defines RoutedLlmClient. Its async call(Request) returns a Response or LlmClientError; the host guide shows the error adapter. Using your own adapter method, as above, also works.

Step::CallDecision

An algorithm can ask a decision model to evaluate a question through Step::CallDecision(call). This time, call holds a CallDecision. It has one model and a DecisionRequest in request.

The algorithm creates the question. The Step loop sends the supplied request through decision_client.call and returns the typed answer.

The adapter returns switchyard_libsy::Result<DecisionResponse>. The optional shared interface is RoutedDecisionClient. The algorithm applies its routing policy to the answer. The decision response is distinct from the final conversational answer.

A decision request contains shared context and named questions. Each question declares its expected answer shape: Boolean, choice among options, or score on a rubric. The client's DecisionResponse uses the same question IDs.

Example payloads: a Boolean question and its answer

These values illustrate what crosses the boundary. The algorithm supplies a request like this in call.request:

let request = DecisionRequest {
    model: Some("Judge".into()),
    context: serde_json::json!({"task": "Fix the failing tests."}),
    questions: BTreeMap::from([(
        "needs_capable".into(),
        DecisionQuestion {
            instructions: serde_json::json!("Does this task need Capable?"),
            kind: DecisionKind::Boolean {
                true_description: None,
                false_description: None,
            },
        },
    )]),
};

The client might return this answer to needs_capable:

let response = DecisionResponse {
    id: None,
    model: Some("Judge".into()),
    answers: BTreeMap::from([(
        "needs_capable".into(),
        DecisionAnswer {
            value: DecisionValue::Boolean(BooleanEstimate::Value(true)),
            provider_confidence: None,
        },
    )]),
    usage: Default::default(),
};

Step::Done

Once the algorithm has enough information, it emits Step::Done(outcome) with a RoutingOutcome. Inside that match arm, check whether routing has already produced an answer:

let response = match outcome.response {
    Some(response) => response, // Reuse the answer obtained during routing.
    None => client.call(outcome.request).await?, // Make the final call.
};

With Some(response), another final call would repeat work already done. With None, your application can send outcome.request now, as above, or return it to a gateway to send later. Signal-only Stage takes this second path.

Use the returned request: it includes algorithm changes and already names the selected model. outcome.selected_model_ids lists that model first, followed by ordered fallbacks.

To read the final answer as text, collect a live stream or use the existing buffered answer:

let answer = response.llm_response.into_agg().await?;
println!("{}", completion_text(&answer));

For live text, set stream: true in the initial LlmRequest before starting the run. Use a streaming-capable adapter; the quickstart's OpenRouter adapter returns buffered responses. Handle both response forms so a buffered answer is still shown:

use std::io::Write;

match response.llm_response {
    LlmResponse::Stream(mut events) => {
        while let Some(event) = events.next().await {
            for chunk in event?.normalized() {
                if let LlmResponseChunk::TextDelta { text, .. } = chunk {
                    print!("{text}");
                    std::io::stdout().flush()?;
                }
            }
        }
    }
    LlmResponse::Agg(answer) => println!("{}", completion_text(&answer)),
}

These are alternative ways to consume a response. The text-only loop omits tool and usage events; handle those chunks too when your application needs them.

outcome.state lets supporting algorithms retain caller-managed state between turns. See Composite sessions for how this differs from a session ID. OutcomeMetadata identifies the run and algorithm and may include evidence for the decision.

Control how work runs

Each call has its own reply channel. Awaiting calls one at a time in the step loop serves them sequentially. To run calls concurrently, your application must keep reading steps while polling their work.

Use the same inputs and clients, but replace the earlier run_stream call and sequential loop with this version. tokio::select! polls both the Step stream and the pending calls, so new calls can start before earlier ones finish:

use futures::{FutureExt, StreamExt, stream::FuturesUnordered};
use switchyard_libsy::LibsyError;

let mut steps = algorithm.run_stream(request, models, state);
let mut in_flight = FuturesUnordered::new();

let outcome = loop {
    tokio::select! {
        Some(result) = in_flight.next() => result?,
        step = steps.next() => {
            let step = step.ok_or(LibsyError::MissingFinalResponse)?;
            match step? {
                Step::CallModel(call) => {
                    let request = call.request.clone();
                    in_flight.push(call.respond(client.call(request)).boxed());
                }
                Step::CallDecision(call) => {
                    let request = call.request.clone();
                    in_flight.push(call.respond(decision_client.call(request)).boxed());
                }
                Step::Done(outcome) => break *outcome,
            }
        }
    }
};
// Stop unused calls before handling the final outcome.
drop(in_flight);
drop(steps);

boxed() gives the two call kinds a common future type for the collection. Only calls requested concurrently by the algorithm can overlap. The returned outcome is handled as shown in Step::Done.

For a ready-made version, see libsy::drive and its reference execution loop.

If the algorithm stops waiting, respond drops that call's work future. The explicit drops stop the remaining handlers and abort the algorithm task. Returning an error or cancelling the enclosing future also drops these local values. Your application must clean up any separately spawned work itself.

A model-call failure normally stops the host run when propagated with ?. call.recover_errors lets the algorithm receive and handle it instead. Decision failures passed through respond also return to the algorithm. Dropping an unanswered call gives the waiting algorithm DriverError::ResponseDropped.

Once a response stream has been delivered, its consumer owns its lifetime. Cancelling local work cannot guarantee that a provider stops generation or billing, or undo side effects that have already occurred. See the host integration guide for Python task cleanup.

The Rust API reference links the complete crate documentation for these types.