Skip to content

managed

managed

Resolve where LLM-assisted PII planning runs, and scope a managed local vLLM server to one planning pass.

Classes:

Name Description
InvalidLocalServerSettingsError

The managed local server's profile, address, or model settings are invalid.

LocalServerRequest

Validated settings for one managed local server launch and the planner that calls it.

Functions:

Name Description
resolve_planning_inference

Resolve where LLM-assisted planning runs, without launching anything.

planning_inference_settings

Yield the inference settings for one LLM planning pass.

Attributes:

Name Type Description
LOCAL_PROFILE_ENV

Bundled profile name or profile YAML path for the managed local vLLM server.

LOCAL_PROFILE_ENV = 'NSS_INFERENCE_LOCAL_PROFILE' module-attribute

Bundled profile name or profile YAML path for the managed local vLLM server.

InvalidLocalServerSettingsError

Bases: ParameterError

The managed local server's profile, address, or model settings are invalid.

LocalServerRequest(profile, host, port, timeout_seconds, request_options=dict()) dataclass

Validated settings for one managed local server launch and the planner that calls it.

Attributes:

Name Type Description
profile LocalVllmProfile

Model and engine settings to serve.

host str

Loopback host the server binds.

port int | None

Port from NSS_INFERENCE_ENDPOINT, or None to choose a free port.

timeout_seconds float

Per-request planner timeout: NSS_INFERENCE_TIMEOUT, else the profile's request_timeout_seconds.

request_options Mapping[str, object]

Chat-completions fields: the profile's, overridden field by field by NSS_INFERENCE_REQUEST_OPTIONS.

profile instance-attribute

Model and engine settings to serve.

host instance-attribute

Loopback host the server binds.

port instance-attribute

Port from NSS_INFERENCE_ENDPOINT, or None to choose a free port.

timeout_seconds instance-attribute

Per-request planner timeout: NSS_INFERENCE_TIMEOUT, else the profile's request_timeout_seconds.

request_options = field(default_factory=dict) class-attribute instance-attribute

Chat-completions fields: the profile's, overridden field by field by NSS_INFERENCE_REQUEST_OPTIONS.

resolve_planning_inference(config, *, environ=None)

Resolve where LLM-assisted planning runs, without launching anything.

Preflight and plan discovery both call this, and it reads the NSS_INFERENCE_* settings through InferenceEnvironment. An explicit NSS_INFERENCE_LOCAL_PROFILE always selects a managed server, and any configured model must match it. Without a profile, an explicit NSS_INFERENCE_ENDPOINT selects that endpoint and no managed server. With neither, the configured model (replace_pii.llm.model_id, then NSS_INFERENCE_MODEL, then DEFAULT_NSS_INFERENCE_LOCAL_MODEL) selects the bundled profile to run.

Parameters:

Name Type Description Default
config LLMConfig

Persisted LLM behavior from replace_pii.llm.

required
environ Mapping[str, str] | None

Environment mapping to read; defaults to os.environ.

None

Returns:

Type Description
LocalServerRequest | InferenceSettings

A LocalServerRequest for a managed local server, or the

LocalServerRequest | InferenceSettings

InferenceSettings for an explicit endpoint.

Raises:

Type Description
InvalidRequestSettingsError

If NSS_INFERENCE_TIMEOUT or NSS_INFERENCE_REQUEST_OPTIONS is invalid.

InvalidLocalServerSettingsError

If the profile is invalid, NSS_INFERENCE_ENDPOINT is not a usable loopback address, a configured model differs from an explicit profile, or no bundled profile serves the model.

MissingInferenceModelError

If an explicit endpoint has no model ID.

ParameterError

If an explicit endpoint is invalid.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/managed.py
def resolve_planning_inference(
    config: LLMConfig,
    *,
    environ: Mapping[str, str] | None = None,
) -> LocalServerRequest | InferenceSettings:
    """Resolve where LLM-assisted planning runs, without launching anything.

    Preflight and plan discovery both call this, and it reads the
    ``NSS_INFERENCE_*`` settings through ``InferenceEnvironment``. An explicit
    ``NSS_INFERENCE_LOCAL_PROFILE`` always selects a managed server, and any
    configured model must match it. Without a profile, an explicit
    ``NSS_INFERENCE_ENDPOINT`` selects that endpoint and no managed server.
    With neither, the configured model (``replace_pii.llm.model_id``, then
    ``NSS_INFERENCE_MODEL``, then ``DEFAULT_NSS_INFERENCE_LOCAL_MODEL``)
    selects the bundled profile to run.

    Args:
        config: Persisted LLM behavior from ``replace_pii.llm``.
        environ: Environment mapping to read; defaults to ``os.environ``.

    Returns:
        A ``LocalServerRequest`` for a managed local server, or the
        ``InferenceSettings`` for an explicit endpoint.

    Raises:
        InvalidRequestSettingsError: If ``NSS_INFERENCE_TIMEOUT`` or
            ``NSS_INFERENCE_REQUEST_OPTIONS`` is invalid.
        InvalidLocalServerSettingsError: If the profile is invalid,
            ``NSS_INFERENCE_ENDPOINT`` is not a usable loopback address, a
            configured model differs from an explicit profile, or no bundled
            profile serves the model.
        MissingInferenceModelError: If an explicit endpoint has no model ID.
        ParameterError: If an explicit endpoint is invalid.
    """
    return _planning_inference(config, InferenceEnvironment.from_environ(environ))

planning_inference_settings(config, *, environ=None)

Yield the inference settings for one LLM planning pass.

For an explicit endpoint (see resolve_planning_inference), this yields its settings. Otherwise it starts a LocalVllmServer, yields settings that point at that server, and stops the server when the block exits. The server therefore never outlives planning, and its GPU memory is free before replacement or training.

Parameters:

Name Type Description Default
config LLMConfig

Persisted LLM behavior from replace_pii.llm.

required
environ Mapping[str, str] | None

Environment mapping to read; defaults to os.environ.

None

Yields:

Type Description
InferenceSettings

Settings the planner sends its requests with.

Raises:

Type Description
ParameterError

If the inference settings are invalid, or the managed server cannot run on this machine.

GenerationError

If the managed server fails to start.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/managed.py
@contextmanager
def planning_inference_settings(
    config: LLMConfig,
    *,
    environ: Mapping[str, str] | None = None,
) -> Iterator[InferenceSettings]:
    """Yield the inference settings for one LLM planning pass.

    For an explicit endpoint (see ``resolve_planning_inference``), this
    yields its settings. Otherwise it starts a ``LocalVllmServer``, yields
    settings that point at that server, and stops the server when the block
    exits. The server therefore never outlives planning, and its GPU memory is
    free before replacement or training.

    Args:
        config: Persisted LLM behavior from ``replace_pii.llm``.
        environ: Environment mapping to read; defaults to ``os.environ``.

    Yields:
        Settings the planner sends its requests with.

    Raises:
        ParameterError: If the inference settings are invalid, or the managed
            server cannot run on this machine.
        GenerationError: If the managed server fails to start.
    """
    runtime_env = os.environ if environ is None else environ
    environment = InferenceEnvironment.from_environ(runtime_env)
    inference = _planning_inference(config, environment)
    if isinstance(inference, InferenceSettings):
        _log_external_destination(inference.endpoint_url)
        yield inference
        return
    if environment.api_key is not None:
        logger.user.warning(
            f"{KEY_ENV} is set, but PII planning runs in a local vLLM server, which uses its own key, so it is "
            f"ignored. Set {ENDPOINT_ENV} to send planning requests to a remote service instead."
        )
    logger.user.info(
        f"PII planning runs {inference.profile.served_name!r} in a local vLLM server; "
        "column samples stay on this machine"
    )
    with LocalVllmServer(
        inference.profile,
        host=inference.host,
        port=inference.port,
        environ=runtime_env,
        request_options=inference.request_options,
    ) as server:
        yield server.inference_settings(max_workers=config.max_workers, timeout_seconds=inference.timeout_seconds)