LLM-Related Protocols

An LLM request describes a conversation, the tools available to the model, and generation settings. Switchyard represents these in LlmRequest. The same type works whether the next call goes to Anthropic Messages, OpenAI Responses, or Chat Completions.

Fields, API mappings, and conversion rules

• Request reference: messages, tools, generation settings, and their provider API fields.
• Response and stream reference: output content, token usage, event types, and stream lifetime.
• Translation reference: engine methods, preservation policies, diagnostics, and edits before encoding.

Read a request

LlmRequest separates instructions from conversation turns. A Message contains a role and ordered ContentBlock values. Content can include text, media, reasoning, tool calls, and tool results.

This example constructs the neutral request passed to an algorithm. It uses the quickstart's dependencies.

use switchyard_protocol::{ContentBlock, InstructionBlock, LlmRequest, Message, Request, Role};

let request = Request {
    llm_request: LlmRequest {
        instructions: vec![InstructionBlock {
            role: Role::System,
            content: vec![ContentBlock::Text { text: "Answer briefly.".into() }],
        }],
        messages: vec![Message::text(Role::User, "Explain tail latency.")],
        ..LlmRequest::default()
    },
    ..Request::default()
};

Request is the host-facing envelope. It adds optional correlation metadata and raw_request, a host-owned JSON body. Codecs use LlmRequest::preservation for replay, not Request::raw_request.

Read a response

The host returns a Response envelope containing LlmResponse. LlmResponse::Agg holds a completed AggLlmResponse. LlmResponse::Stream holds a single-consumption stream of LlmResponseStreamEvent values. Each stream event exposes neutral chunks through normalized(). Chunks describe message starts, text deltas, tool arguments, usage, and completion. An event may also retain parsed provider JSON for same-format replay.

Usage::input_tokens counts non-cached input. The codecs separate cache-read and cache-write tokens from OpenAI's aggregate input counts. Missing counts stay None. Call LlmResponse::into_agg().await when you need a buffered result; this consumes the stream and aggregates its normalized chunks.

Understand lossless conversion

The protocol retains source data for lossless same-format round trips. With the default preservation policy, decoding a supported API body and encoding it back to that API reuses its preserved JSON. The preservation reference explains exactly what is retained and when replay takes precedence.

Conversion between different provider APIs can be lossy. For example, Anthropic has no separate developer role, and Chat Completions has no native equivalent for every Responses built-in tool. Keeping the original data in the protocol does not give the destination API those capabilities.

Translation reports known losses in diagnostics. The translation policy reference explains how to allow or reject these conversions.

Encoding an edited request

A preserved same-format body takes precedence over normalized fields. If you change the protocol fields and want those edits encoded, remove that format's entry from preservation.requests, or encode with PreservationPolicy::Disabled. The same rule applies to preservation.responses for buffered responses.

See stream preservation and aggregation for how replay and edits work on streamed responses.

Use the translation crate

switchyard-translation supplies the LLM codecs without making network calls. Add it at the same API revision as libsy and switchyard-protocol, following the quickstart's translation setup.

This excerpt decodes an inbound Chat Completions body and encodes it as an OpenAI Responses request. It can run inside your adapter; outbound.body is the JSON your client sends.

use serde_json::json;
use switchyard_translation::{TranslationEngine, TranslationPolicy, WireFormat};

let engine = TranslationEngine::default();
let policy = TranslationPolicy::default();
let inbound = json!({
    "model": "your-model",
    "messages": [{"role": "user", "content": "Explain tail latency."}]
});

let decoded = engine.decode_request(WireFormat::OpenAiChat, &inbound, &policy)?;
let outbound = engine.encode_request(
    WireFormat::OpenAiResponses,
    &decoded.request,
    &policy,
)?;
// Check diagnostics from both operations before sending the encoded body.

During a libsy run, place decoded.request in Request::llm_request. Encode the request from Step::CallModel for the selected backend. Decode its reply with decode_response, or use decode_stream_event and encode_stream_event with a separate StreamTranslationState per response stream. Return the neutral Response through the call object, as shown in Mechanics of Libsy.

The engine also exposes translate_request and translate_response for direct API-to-API conversion. See the translation engine reference for signatures and results.

Keep your own client or conversions

You can construct these protocol types from your existing SDK and convert them back yourself. When handling Step::CallModel, pass call.respond a future that uses your client and returns libsy::Result<Response>. Keep content order, tool IDs, provider-only data, and streaming lifetime intact where your integration needs them.

RoutedLlmClient is an optional shared interface used by Switchyard's clients and execution helpers. Direct run_stream handling works with your own functions or clients; it does not require implementing that trait.

If you want Switchyard to handle HTTP and translation, use TranslatingLlmClient from the HTTP client crate. The quickstart shows how to keep your own HTTP client. Typed decisions use the separate contract described in Decision Model-Related Protocols.