Host Integration

Add libsy where your gateway or agent harness chooses and calls a model. Your application keeps its transport, credentials, and lifecycle.

Start with the Rust quickstart for a working example. Use Mechanics of Libsy for execution and Switchyard Protocols for data conversion. This page covers the responsibilities specific to your host.

Choose the integration boundary

Host	Insert routing here	Preserve from the existing application
Gateway	After request admission and input decoding, before model dispatch.	Authentication, tenant isolation, budgets, model inventory, provider credentials, and response delivery.
Agent harness	Before each model turn, after assembling conversation and tool history.	Tool execution, task state, model clients, conversation storage, and the agent loop.

Host responsibilities

Dispatch every requested call. Apply your credentials, budgets, and access policy to routing-time judge calls as well as answer calls. Send internal calls through model dispatch without entering the same router again. Supply only models your host can reach. See call handling and final-outcome handling.

Keep history and state. Supply the tool history required by the algorithm; provider continuation IDs alone can hide that evidence. Scope session IDs to the tenant and conversation. Reuse algorithm instances when they retain session decisions, and decide how your host handles concurrent turns and multiple processes. State behavior is algorithm-specific; see Composite sessions and the Plan/Execute history requirement.

Observe the complete run. Include judge calls and final model work in your host's observations. Configure telemetry collection in your application and instrument model I/O performed by your own clients. See the OpenTelemetry reference.

For existing integrations, check NeMo Relay or the LiteLLM compatibility notes. The experimental LiteLLM plugin supports decision-only routing and cannot serve intermediate model calls.

Serve routing-time calls

Use your existing client as shown in the Step loop. The shared client traits below are optional.

Optional: adapt the shared client traits

Development API

These adapters use the quickstart's development dependencies. Their asynchronous reply method differs from the synchronous callback API in the released v0.3.0 crates.

When using RoutedLlmClient or RoutedDecisionClient, adapt errors to LibsyError:

use switchyard_libsy::{CallDecision, CallModel, LibsyError, Result};
use switchyard_protocol::{RoutedDecisionClient, RoutedLlmClient};

async fn serve(call: CallModel, client: &dyn RoutedLlmClient) -> Result<()> {
    let request = call.request.clone();
    let model = call.models[0].clone();
    call.respond(async move {
        client.call(request).await.map_err(|error| LibsyError::client_call(model, error))
    }).await
}

async fn serve_decision(call: CallDecision, client: &dyn RoutedDecisionClient) -> Result<()> {
    let request = call.request.clone();
    let model = call.model.clone();
    call.respond(async move {
        client.call(request).await.map_err(|error| LibsyError::client_call(model, error))
    }).await
}

Python bindings

The bindings run the same Rust algorithms. The local routing example below works with the released package. For the asynchronous callbacks and cleanup described after it, build matching development bindings and follow the complete Python host example.

Python setup, callbacks, and metadata

With Python 3.10+ and uv, create a project and install the released package:

uv init --bare switchyard-python-demo
cd switchyard-python-demo
uv add "nemo-switchyard==0.3.0"

Save as route.py:

import asyncio
from switchyard.libsy import Step, algorithms

async def main():
    router = algorithms.stage_router(
        picker="efficient_first", confidence_threshold=0.5)
    models = {"efficient": ["Efficient"], "capable": ["Capable"]}
    models["any"] = ["Efficient", "Capable"]
    text = {"type": "text", "text": "Hello"}
    request = {"messages": [{"role": "user", "content": [text]}]}
    async for step in router.run_stream(request, models):
        if isinstance(step, Step.Done):
            print("Selected:", step.outcome.selected_model_ids[0])

asyncio.run(main())

uv run route.py prints Selected: Efficient without making a network call.

Concurrent calls and cancellation

The client coroutine returns LlmResponse.Agg or LlmResponse.Stream. Pass the coroutine itself to respond:

await call.respond(client.call(call.request, call.models[0]))

respond cancels and awaits the supplied provider work before returning. For concurrent algorithms, create handler tasks and cancel and await the remaining handlers when the run ends, as in the Python host example. Awaiting one handler at a time serializes execution. call.fail(error) is also awaitable; decision calls use the same reply pattern.

For the shared cancellation behavior and upstream limits, see Mechanics.

Metadata and telemetry

After checking that outcome.metadata is present, read its outcome_id, algorithm, and evidence attributes. Evidence is a normal Python value, usually a dictionary; algorithms without evidence return None. See the OpenTelemetry reference for signals and export setup.