pub struct Driver { /* private fields */ }Expand description
The offload channel handed to an algorithm’s
create_run_task. The algorithm makes model calls
with call_llm_target (or call_llm) and
publishes its Decisions with info; each call is offloaded to the
request’s Step stream and awaits the consumer’s response. The step channel is
bounded, so the consumer paces the algorithm one step at a time.
Implementations§
Source§impl Driver
impl Driver
Sourcepub async fn call_llm(&self, routed: RoutedRequest) -> Result<Response>
pub async fn call_llm(&self, routed: RoutedRequest) -> Result<Response>
Offload a model call: publish routed as a Step::CallLlm and await the
consumer’s Response. The call’s context travels inside
routed.ctx. Errors if the stream is closed or the call failed.
The await is wrapped in a libsy.llm_call span measuring fulfillment as
the algorithm observes it (host queueing/serving included; a streamed
response resolves when its stream handle arrives); latency, outcome, and
token usage are recorded when it resolves. The provider call itself gets a
libsy.client_call span when Algorithm::run serves it.
Sourcepub async fn call_llm_target(
&self,
ctx: Context,
target: &LlmTarget,
request: Request,
decision: Arc<dyn Decision>,
) -> Result<Response>
pub async fn call_llm_target( &self, ctx: Context, target: &LlmTarget, request: Request, decision: Arc<dyn Decision>, ) -> Result<Response>
Offload a call to target: pair request with decision and the target’s
default client into a RoutedRequest, then publish it (see
call_llm). The convenience most algorithms use;
decision.selected_model() names the model to hit, and request’s
model is left untouched.