pub struct TranslatingLlmClient { /* private fields */ }Expand description
A client that dispatches neutral-IR requests to named targets.
Each ModelConfig defines a target’s provider model and backends.
Each call resolves the target and wire format, encodes the request,
applies auth and forwarded headers, sends the HTTP request with a shared
[reqwest::Client], and decodes the response back to the neutral IR (buffered
or streamed).
Implementations§
Source§impl TranslatingLlmClient
impl TranslatingLlmClient
Sourcepub fn new(model_configs: &[ModelConfig]) -> Result<Self>
pub fn new(model_configs: &[ModelConfig]) -> Result<Self>
Builds a client over the given ModelConfigs, with a fresh shared HTTP
client and the built-in translation codecs.
Sourcepub fn without_rate_limit_cooldown(self) -> Self
pub fn without_rate_limit_cooldown(self) -> Self
Disable HTTP 429 cooldown so subsequent calls can receive the upstream error.
Sourcepub fn backend_for(
&self,
target_name: &ModelId,
format: WireFormat,
) -> Option<&Backend>
pub fn backend_for( &self, target_name: &ModelId, format: WireFormat, ) -> Option<&Backend>
The target’s backend for format, or None if no matching backend exists.
Sourcepub fn provider_model<'a>(&'a self, target_name: &'a ModelId) -> &'a ModelId
pub fn provider_model<'a>(&'a self, target_name: &'a ModelId) -> &'a ModelId
The target’s provider model ID, falling back to its name when none is configured.
Sourcepub fn supports_auxiliary(
&self,
target_name: &ModelId,
operation: AuxiliaryOperation,
) -> bool
pub fn supports_auxiliary( &self, target_name: &ModelId, operation: AuxiliaryOperation, ) -> bool
Whether the target has a backend for operation.
Sourcepub async fn get_model_properties(
&self,
target_name: &ModelId,
format: WireFormat,
timeout: Duration,
) -> Result<Value>
pub async fn get_model_properties( &self, target_name: &ModelId, format: WireFormat, timeout: Duration, ) -> Result<Value>
Reads llama.cpp model properties through the backend’s configured auth and headers.
Redirects are disabled so credentials cannot move to another origin.
Sourcepub async fn call_auxiliary(
&self,
target_name: &ModelId,
request: Request,
operation: AuxiliaryOperation,
) -> Result<Value>
pub async fn call_auxiliary( &self, target_name: &ModelId, request: Request, operation: AuxiliaryOperation, ) -> Result<Value>
Calls a model-bearing auxiliary provider operation.
Returns an error when the model has no compatible backend or the upstream request fails or returns invalid JSON.
Sourcepub async fn call_rewrite_model(
&self,
request: Request,
target_name: Option<&ModelId>,
) -> Result<Response>
pub async fn call_rewrite_model( &self, request: Request, target_name: Option<&ModelId>, ) -> Result<Response>
Calls the selected target using its configured provider model and backend.
target_name takes precedence over the request’s model field.
Request metadata selects the wire format, falling back to the target’s default.