managed
managed
¶
Resolve where LLM-assisted PII planning runs, and scope a managed local vLLM server to one planning pass.
Classes:
| Name | Description |
|---|---|
InvalidLocalServerSettingsError |
The managed local server's profile, address, or model settings are invalid. |
LocalServerRequest |
Validated settings for one managed local server launch and the planner that calls it. |
Functions:
| Name | Description |
|---|---|
resolve_planning_inference |
Resolve where LLM-assisted planning runs, without launching anything. |
planning_inference_settings |
Yield the inference settings for one LLM planning pass. |
Attributes:
| Name | Type | Description |
|---|---|---|
LOCAL_PROFILE_ENV |
Bundled profile name or profile YAML path for the managed local vLLM server. |
LOCAL_PROFILE_ENV = 'NSS_INFERENCE_LOCAL_PROFILE'
module-attribute
¶
Bundled profile name or profile YAML path for the managed local vLLM server.
InvalidLocalServerSettingsError
¶
Bases: ParameterError
The managed local server's profile, address, or model settings are invalid.
LocalServerRequest(profile, host, port, timeout_seconds, request_options=dict())
dataclass
¶
Validated settings for one managed local server launch and the planner that calls it.
Attributes:
| Name | Type | Description |
|---|---|---|
profile |
LocalVllmProfile
|
Model and engine settings to serve. |
host |
str
|
Loopback host the server binds. |
port |
int | None
|
Port from |
timeout_seconds |
float
|
Per-request planner timeout: |
request_options |
Mapping[str, object]
|
Chat-completions fields: the profile's, overridden field by field by |
profile
instance-attribute
¶
Model and engine settings to serve.
host
instance-attribute
¶
Loopback host the server binds.
port
instance-attribute
¶
Port from NSS_INFERENCE_ENDPOINT, or None to choose a free port.
timeout_seconds
instance-attribute
¶
Per-request planner timeout: NSS_INFERENCE_TIMEOUT, else the profile's request_timeout_seconds.
request_options = field(default_factory=dict)
class-attribute
instance-attribute
¶
Chat-completions fields: the profile's, overridden field by field by NSS_INFERENCE_REQUEST_OPTIONS.
resolve_planning_inference(config, *, environ=None)
¶
Resolve where LLM-assisted planning runs, without launching anything.
Preflight and plan discovery both call this, and it reads the
NSS_INFERENCE_* settings through InferenceEnvironment. An explicit
NSS_INFERENCE_LOCAL_PROFILE always selects a managed server, and any
configured model must match it. Without a profile, an explicit
NSS_INFERENCE_ENDPOINT selects that endpoint and no managed server.
With neither, the configured model (replace_pii.llm.model_id, then
NSS_INFERENCE_MODEL, then DEFAULT_NSS_INFERENCE_LOCAL_MODEL)
selects the bundled profile to run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
LLMConfig
|
Persisted LLM behavior from |
required |
environ
|
Mapping[str, str] | None
|
Environment mapping to read; defaults to |
None
|
Returns:
| Type | Description |
|---|---|
LocalServerRequest | InferenceSettings
|
A |
LocalServerRequest | InferenceSettings
|
|
Raises:
| Type | Description |
|---|---|
InvalidRequestSettingsError
|
If |
InvalidLocalServerSettingsError
|
If the profile is invalid,
|
MissingInferenceModelError
|
If an explicit endpoint has no model ID. |
ParameterError
|
If an explicit endpoint is invalid. |
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/managed.py
planning_inference_settings(config, *, environ=None)
¶
Yield the inference settings for one LLM planning pass.
For an explicit endpoint (see resolve_planning_inference), this
yields its settings. Otherwise it starts a LocalVllmServer, yields
settings that point at that server, and stops the server when the block
exits. The server therefore never outlives planning, and its GPU memory is
free before replacement or training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
LLMConfig
|
Persisted LLM behavior from |
required |
environ
|
Mapping[str, str] | None
|
Environment mapping to read; defaults to |
None
|
Yields:
| Type | Description |
|---|---|
InferenceSettings
|
Settings the planner sends its requests with. |
Raises:
| Type | Description |
|---|---|
ParameterError
|
If the inference settings are invalid, or the managed server cannot run on this machine. |
GenerationError
|
If the managed server fails to start. |