Trait RoutedLlmClient 

pub trait RoutedLlmClient: Send + Sync {
    // Required method
    fn call<'life0, 'async_trait>(
        &'life0 self,
        ctx: Context,
        request: Request,
        decision: Arc<dyn Decision>,
    ) -> Pin<Box<dyn Future<Output = Result<Response, LlmClientError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided methods
    fn supports_count_tokens(&self) -> bool { ... }
    fn count_tokens<'life0, 'async_trait>(
        &'life0 self,
        request: Request,
    ) -> Pin<Box<dyn Future<Output = Result<Value, LlmClientError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait { ... }
}

Performs the actual model call for a target. This is the one piece of I/O the library does not own — a host implements it over its own transport (HTTP SDK, in-process model, mock). It serves a call the stream consumer chose not to override, reached as a routed request’s default_client.

Concurrency

A client may be shared by many targets and concurrent algorithm runs. Calls may overlap, so implementations must synchronize mutable state internally and should not serialize requests unless their transport requires it.

Required Methods

fn call<'life0, 'async_trait>( &'life0 self, ctx: Context, request: Request, decision: Arc<dyn Decision>, ) -> Pin<Box<dyn Future<Output = Result<Response, LlmClientError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Serve the call, returning the model’s response. Call the model named by decision.selected_model() — the target the algorithm routed to — mapping it to whatever provider model id this client hits. request.llm_request.model is the agent’s original name, carried through for reference, not a call target. ctx carries the request’s cross-cutting state.

Provided Methods

fn supports_count_tokens(&self) -> bool

Whether this client can serve count_tokens — i.e. it has an Anthropic upstream. The default is false.

fn count_tokens<'life0, 'async_trait>( &'life0 self, request: Request, ) -> Pin<Box<dyn Future<Output = Result<Value, LlmClientError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Count the tokens request would use — a direct passthrough, not a routed call. Forwards request to this client’s Anthropic /v1/messages/count_tokens endpoint (model restamped to the upstream target id) and returns the JSON verbatim. Token counting is a pre-flight estimate with no routing decision, so unlike call it takes no Decision. The default errors; only an Anthropic-backed client overrides it.

Implementors