Struct TranslatingLlmClient 

pub struct TranslatingLlmClient { /* private fields */ }

A client that dispatches neutral-IR requests to named targets.

Each ModelConfig defines a target’s provider model and backends. Each call resolves the target and wire format, encodes the request, applies auth and forwarded headers, sends the HTTP request with a shared [reqwest::Client], and decodes the response back to the neutral IR (buffered or streamed).

Implementations

impl TranslatingLlmClient

pub fn new(model_configs: &[ModelConfig]) -> Result<Self>

Builds a client over the given ModelConfigs, with a fresh shared HTTP client and the built-in translation codecs.

pub fn without_rate_limit_cooldown(self) -> Self

Disable HTTP 429 cooldown so subsequent calls can receive the upstream error.

pub fn backend_for( &self, target_name: &ModelId, format: WireFormat, ) -> Option<&Backend>

The target’s backend for format, or None if no matching backend exists.

pub fn provider_model<'a>(&'a self, target_name: &'a ModelId) -> &'a ModelId

The target’s provider model ID, falling back to its name when none is configured.

pub fn supports_auxiliary( &self, target_name: &ModelId, operation: AuxiliaryOperation, ) -> bool

Whether the target has a backend for operation.

pub async fn get_model_properties( &self, target_name: &ModelId, format: WireFormat, timeout: Duration, ) -> Result<Value>

Reads llama.cpp model properties through the backend’s configured auth and headers.

Redirects are disabled so credentials cannot move to another origin.

pub async fn call_auxiliary( &self, target_name: &ModelId, request: Request, operation: AuxiliaryOperation, ) -> Result<Value>

Calls a model-bearing auxiliary provider operation.

Returns an error when the model has no compatible backend or the upstream request fails or returns invalid JSON.

pub async fn call_rewrite_model( &self, request: Request, target_name: Option<&ModelId>, ) -> Result<Response>

Calls the selected target using its configured provider model and backend. target_name takes precedence over the request’s model field. Request metadata selects the wire format, falling back to the target’s default.

pub async fn call_rewrite_model_raw( &self, raw_http_request: Value, http_headers: Option<HeaderMap>, target_name: Option<&ModelId>, wire_format: WireFormat, ) -> Result<RawResponse>

The whole decode → call → encode path a wire endpoint needs, in one call.

Decodes raw_http_request from wire_format to the neutral IR, serves it via call_rewrite_model — the upstream wire format is resolved there from the model’s backend, independently of wire_format — then encodes the neutral response back into wire_format. The result is a buffered RawResponse::Buffered JSON body or a streamed RawResponse::Stream of wire events (the caller frames the stream as SSE). The response’s model is restamped with the model that actually served the call, so the body names the model that answered rather than the route the caller addressed.

http_headers are carried through as the request’s Metadata::http_headers. Backends with forward_auth disabled forward only allowed metadata headers; forward_auth backends forward all application headers. The forwarding backends in one route must use one credential family unless they all use the same scheme, host, and port. Transport headers are always rebuilt. Pass None to forward nothing.

Trait Implementations

impl RoutedLlmClient for TranslatingLlmClient

fn provider_model<'a>(&'a self, target: &'a ModelId) -> &'a ModelId

Resolves a routing key for telemetry. Direct model clients keep the key unchanged.

fn call<'life0, 'async_trait>( &'life0 self, request: Request, ) -> Pin<Box<dyn Future<Output = Result<Response>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Make a request