Struct TranslatingLlmClient 

pub struct TranslatingLlmClient { /* private fields */ }

A client that dispatches neutral-IR requests to per-model HTTP backends.

Construct it with a list of ModelConfigs — one per model, each naming a default Backend and any additional per-format backends. Each call resolves the model and wire format, encodes the request to that backend’s wire format, applies auth and forwarded headers, sends the HTTP request with a shared [reqwest::Client], and decodes the response back to the neutral IR (buffered or streamed).

Implementations

impl TranslatingLlmClient

pub fn new(model_configs: &[ModelConfig]) -> Result<Self>

Builds a client over the given ModelConfigs, with a fresh shared HTTP client and the built-in translation codecs.

pub fn without_rate_limit_cooldown(self) -> Self

Disable HTTP 429 cooldown so subsequent calls can receive the upstream error.

pub fn backend_for( &self, model: &ModelId, format: WireFormat, ) -> Option<&Backend>

The backend serving model over format — the default backend when its format matches, otherwise a matching entry in other_backends; None when the model is unknown or has no backend for format.

pub fn supports_auxiliary( &self, model: &ModelId, operation: AuxiliaryOperation, ) -> bool

Whether model has a backend for operation.

pub async fn get_model_properties( &self, model: &ModelId, format: WireFormat, timeout: Duration, ) -> Result<Value>

Reads llama.cpp model properties through the backend’s configured auth and headers.

Redirects are disabled so credentials cannot move to another origin.

pub async fn call_auxiliary( &self, model: &ModelId, request: Request, operation: AuxiliaryOperation, ) -> Result<Value>

Calls a model-bearing auxiliary provider operation.

Returns an error when the model has no compatible backend or the upstream request fails or returns invalid JSON.

pub async fn call_rewrite_model( &self, request: Request, model_name: Option<&ModelId>, ) -> Result<Response>

Calls the backend for model_name (or the request’s own model), over the wire format the request pins in its metadata (else the model’s default backend), and returns the neutral response.

Resolution: model_name wins over request.llm_request.model; the resolved name is both the outer map key and the model id written into the request before translation. Missing models are invalid requests; unknown models or wire formats are configuration errors.

pub async fn call_rewrite_model_raw( &self, raw_http_request: Value, http_headers: Option<HeaderMap>, model: Option<&ModelId>, wire_format: WireFormat, ) -> Result<RawResponse>

The whole decode → call → encode path a wire endpoint needs, in one call.

Decodes raw_http_request from wire_format to the neutral IR, serves it via call_rewrite_model — the upstream wire format is resolved there from the model’s backend, independently of wire_format — then encodes the neutral response back into wire_format. The result is a buffered RawResponse::Buffered JSON body or a streamed RawResponse::Stream of wire events (the caller frames the stream as SSE). The response’s model is restamped with the model that actually served the call, so the body names the model that answered rather than the route the caller addressed.

http_headers are carried through as the request’s Metadata::http_headers. Backends with forward_auth disabled forward only allowed metadata headers; forward_auth backends forward all application headers. The forwarding backends in one route must use one credential family unless they all use the same scheme, host, and port. Transport headers are always rebuilt. Pass None to forward nothing.

Trait Implementations

impl RoutedLlmClient for TranslatingLlmClient

fn call<'life0, 'async_trait>( &'life0 self, request: Request, ) -> Pin<Box<dyn Future<Output = Result<Response>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Make a request