LLM Responses and Streams

For the integration flow, read LLM-Related Protocols. For the input side, use LLM Requests.

Response forms

Response is the host-facing envelope. Its llm_response is either LlmResponse::Agg, containing an AggLlmResponse, or LlmResponse::Stream, containing a live LlmResponseStream. The stream has one consumer.

Protocol field and type	Meaning	Anthropic Messages	OpenAI Responses	Chat Completions
AggLlmResponse.id: Option<String>	Provider response identifier, if available.	id	id	id
AggLlmResponse.model: Option<String>	Provider-reported model name.	model	model	model
AggLlmResponse.outputs: Vec<ResponseOutput>	Completed output items.	Assistant content blocks	Items in output	choices[].message
ResponseOutput.content: Vec<ContentBlock>	Ordered text, reasoning, tool, and other output content.	text, thinking, tool_use, and other blocks	Message content, reasoning, function calls, and other items	Message content, tool_calls, and provider-specific reasoning fields
ResponseOutput.stop_reason: Option<StopReason>	Why this output stopped.	stop_reason	Response status and incomplete_details	choices[].finish_reason
ResponseOutput.url_citations: Vec<UrlCitation>	Cited URLs with text offsets.	Not normalized by the current codec	output_text.annotations	Message annotations when supplied
AggLlmResponse.usage: Usage	Normalized usage counts and any provider-reported cost.	usage	usage	usage
AggLlmResponse.extensions: ProviderExtensions	Provider-only response fields.	Source fields retained by the codec	Source fields retained by the codec	Source fields retained by the codec
AggLlmResponse.preservation: PreservationMetadata	Original JSON for same-format buffered replay.	Original parsed body	Original parsed body	Original parsed body

The envelope also carries Response.metadata and Response.upstream_headers. Response.served_model() reads the Switchyard target recorded in metadata. Response.selected_model() reads the provider model from a buffered result and returns no model for a live stream, since inspecting its events would consume it.

Usage

Usage.input_tokens counts non-cached input. When an OpenAI-style provider reports aggregate input counts, the codecs subtract reported cache-read and cache-write counts. The cache details remain separate.

Protocol field and type	Meaning	Anthropic Messages	OpenAI Responses	Chat Completions
Usage.input_tokens: Option<u64>	Non-cached input tokens.	input_tokens	input_tokens minus reported cache detail	prompt_tokens minus reported cache detail
InputCacheUsage.cached_input_tokens: Option<u64>	Input read from cache.	cache_read_input_tokens	Input token details, when reported	Prompt token details, when reported
InputCacheUsage.cache_creation_input_tokens: Option<u64>	Input written to cache.	cache_creation_input_tokens	Input token details, when reported	Prompt token details, when reported
Usage.output_tokens: Option<u64>	Output tokens, including reasoning when counted by the provider.	output_tokens	output_tokens	completion_tokens
Usage.reasoning_tokens: Option<u64>	Reasoning output reported separately.	Provider token details, when reported	output_tokens_details.reasoning_tokens	Completion token details, when reported
Usage.total_tokens: Option<u64>	Provider-reported or codec-computed total.	Computed when needed	total_tokens or computed total	total_tokens or computed total
Usage.provider_cost: Option<f64>	Provider-reported monetary charge in USD.	Only when reported	Only when reported	Only when reported

Missing counts remain unknown. Do not turn an absent count into a measured zero when adapting a response. Usage.cache holds optional InputCacheUsage details.

Stream events

Each item of LlmResponseStream is a result containing LlmResponseStreamEvent. An event contains zero or more neutral LlmResponseChunk values and may retain one parsed provider event for same-format replay.

Protocol variant	Meaning	Anthropic Messages	OpenAI Responses	Chat Completions
LlmResponseChunk::MessageStart	Establish or update response identity.	message_start	Response creation/start events	Initial model and response fields
LlmResponseChunk::TextDelta	Append text at an output index.	Text content_block_delta	Output text delta events	choices[].delta.content
LlmResponseChunk::ReasoningDelta	Append reasoning text.	Thinking deltas	Reasoning summary deltas	Provider-specific reasoning deltas
LlmResponseChunk::ReasoningDetailsDelta	Retain structured or encrypted reasoning detail.	Reasoning signatures and related detail	Encrypted reasoning items	Provider-specific structured reasoning fields
LlmResponseChunk::ToolCallDelta	Build a tool call's ID, name, and argument string.	Tool start and input JSON deltas	Function-call item and argument events	Tool-call deltas
LlmResponseChunk::Usage	Update available token counts.	Message usage and deltas	Usage on response events	Usage chunks
LlmResponseChunk::MessageStop	End the message.	Message stop events	Response completion events	Finish reasons and terminal framing
LlmResponseChunk::DecodeError / LlmResponseChunk::StreamError	Report decoding or in-band upstream failure.	Codec/upstream error events	Codec/upstream error events	Codec/upstream error events

These are semantic mappings. One provider event can produce several chunks, and some events have no normalized chunks. Raw replay can still retain those events. Algorithms read neutral chunks through normalized().

Stream lifetime and preservation

Pass original LlmResponseStreamEvent values through when exact same-format parsed-JSON replay matters. replace_normalized() returns an event with the new chunks and clears its stale raw replay data.

LlmResponse::into_agg().await consumes the stream and folds its normalized chunks into an AggLlmResponse. Stream item failures, decode failures, and in-band upstream errors become errors. Aggregation does not preserve the original provider event sequence, MCP activity, unknown blocks, or all response extensions. The current accumulator also folds output indices into one assistant output.

Dropping a stream stops local consumption. It cannot guarantee that the provider stops generation or billing. See concurrency and cancellation for host-owned work, and Translation for decoding, encoding, and finishing provider streams.