Skip to content

llm_client

llm_client

Shared OpenAI-compatible client for PII LLM operations.

Classes:

Name Description
InvalidRequestSettingsError

NSS_INFERENCE_TIMEOUT or NSS_INFERENCE_REQUEST_OPTIONS is invalid.

InferenceSettings

Resolved runtime settings for the OpenAI-compatible adapter.

TransientInferenceError

Retryable inference transport failure without response content.

InvalidInferenceResponse

Retryable malformed inference envelope without response content.

MissingInferenceModelError

An explicit endpoint is configured, but no model ID for it.

LLMTransport

Transport capable of requesting a structured LLM response.

OpenAICompatibleTransport

Minimal privacy-preserving OpenAI chat-completions transport.

Functions:

Name Description
nonblank

Return value stripped of surrounding whitespace, or None if blank.

is_loopback_host

Return whether hostname is localhost or a loopback IP address.

merge_request_options

Merge request-option layers field by field, dropping fields set to None.

reserved_request_fields

Return the sorted fields in fields that NSS sets itself.

configured_model_id

Return config.model_id, else NSS_INFERENCE_MODEL, or None when neither is set.

resolve_inference_settings

Resolve settings for an explicit inference endpoint from configuration and environment.

endpoint_inference_settings

Return settings for the explicit endpoint in environment; see resolve_inference_settings.

Attributes:

Name Type Description
ENDPOINT_ENV

OpenAI-compatible inference endpoint; when set, planning sends requests there instead of a local server.

KEY_ENV

API key for the endpoint.

MODEL_ENV

Model ID when replace_pii.llm.model_id is unset.

REQUEST_OPTIONS_ENV

Extra chat-completions fields, as a JSON object.

RESERVED_REQUEST_FIELDS

Chat-completions fields NSS sets itself and request options must not override.

TIMEOUT_ENV

Per-request timeout in seconds.

ENDPOINT_ENV = 'NSS_INFERENCE_ENDPOINT' module-attribute

OpenAI-compatible inference endpoint; when set, planning sends requests there instead of a local server.

KEY_ENV = 'NSS_INFERENCE_KEY' module-attribute

API key for the endpoint.

MODEL_ENV = 'NSS_INFERENCE_MODEL' module-attribute

Model ID when replace_pii.llm.model_id is unset.

REQUEST_OPTIONS_ENV = 'NSS_INFERENCE_REQUEST_OPTIONS' module-attribute

Extra chat-completions fields, as a JSON object.

RESERVED_REQUEST_FIELDS = frozenset({'messages', 'model', 'response_format', 'stream'}) module-attribute

Chat-completions fields NSS sets itself and request options must not override.

TIMEOUT_ENV = 'NSS_INFERENCE_TIMEOUT' module-attribute

Per-request timeout in seconds.

InvalidRequestSettingsError

Bases: ParameterError

NSS_INFERENCE_TIMEOUT or NSS_INFERENCE_REQUEST_OPTIONS is invalid.

InferenceSettings(endpoint_url, model_id, max_workers, timeout_seconds=DEFAULT_NSS_INFERENCE_TIMEOUT_SECONDS, request_options=dict(), api_key=None) dataclass

Resolved runtime settings for the OpenAI-compatible adapter.

Attributes:

Name Type Description
endpoint_url str

Chat-completions base URL without a trailing slash.

model_id str

Model identifier sent with every request.

max_workers int

Maximum concurrent requests.

timeout_seconds float

Per-request timeout; reasoning models on local GPUs can need several minutes.

request_options Mapping[str, object]

Extra chat-completions fields, such as sampling settings, sent with every request.

api_key str | None

Bearer credential, or None for keyless local endpoints; excluded from repr.

endpoint_url instance-attribute

Chat-completions base URL without a trailing slash.

model_id instance-attribute

Model identifier sent with every request.

max_workers instance-attribute

Maximum concurrent requests.

timeout_seconds = DEFAULT_NSS_INFERENCE_TIMEOUT_SECONDS class-attribute instance-attribute

Per-request timeout; reasoning models on local GPUs can need several minutes.

request_options = field(default_factory=dict) class-attribute instance-attribute

Extra chat-completions fields, such as sampling settings, sent with every request.

Fields left out, including temperature, use the server's defaults; vLLM takes them from the model's generation_config.json.

api_key = field(default=None, repr=False) class-attribute instance-attribute

Bearer credential, or None for keyless local endpoints; excluded from repr.

TransientInferenceError(message, *, retry_after=None)

Bases: GenerationError

Retryable inference transport failure without response content.

Parameters:

Name Type Description Default
message str

Description of the failure; must not contain response bodies.

required
retry_after float | None

Server-requested delay in seconds from a numeric Retry-After header, or None when absent.

None
Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def __init__(self, message: str, *, retry_after: float | None = None) -> None:
    super().__init__(message)
    self.retry_after = retry_after

InvalidInferenceResponse

Bases: GenerationError

Retryable malformed inference envelope without response content.

MissingInferenceModelError

Bases: ParameterError

An explicit endpoint is configured, but no model ID for it.

LLMTransport

Bases: Protocol

Transport capable of requesting a structured LLM response.

Methods:

Name Description
complete

Return the assistant response text without logging it.

complete(*, messages, response_model)

Return the assistant response text without logging it.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def complete(
    self,
    *,
    messages: Sequence[Mapping[str, str]],
    response_model: type[BaseModel],
) -> str:
    """Return the assistant response text without logging it."""

OpenAICompatibleTransport(settings)

Minimal privacy-preserving OpenAI chat-completions transport.

Parameters:

Name Type Description Default
settings InferenceSettings

Resolved endpoint, model, credential, timeout, and request options.

required

Raises:

Type Description
ParameterError

If the endpoint is malformed, embeds credentials, or uses plaintext HTTP for a non-loopback host.

Methods:

Name Description
complete

Return one structured assistant response.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def __init__(self, settings: InferenceSettings) -> None:
    # Settings built directly, not through ``resolve_inference_settings``,
    # must not send samples or the bearer key over an unsafe endpoint.
    _validate_endpoint(settings.endpoint_url)
    self._settings = settings
    # Loopback servers never need a proxy, and an environment proxy would
    # receive the bearer key and raw cell samples. Remote endpoints keep
    # the environment's proxy settings.
    self._trust_env = not is_loopback_host(urlparse(settings.endpoint_url).hostname)

complete(*, messages, response_model)

Return one structured assistant response.

Errors intentionally exclude response bodies because they may contain prompts, samples, or model-authored sensitive text.

Parameters:

Name Type Description Default
messages Sequence[Mapping[str, str]]

Chat messages to send.

required
response_model type[BaseModel]

Pydantic model whose strict JSON schema constrains the response.

required

Returns:

Type Description
str

The assistant message content.

Raises:

Type Description
ParameterError

On authentication, authorization, or other permanent request rejection.

TransientInferenceError

On network failures, timeouts, or retryable HTTP statuses.

InvalidInferenceResponse

If the response envelope is malformed.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def complete(
    self,
    *,
    messages: Sequence[Mapping[str, str]],
    response_model: type[BaseModel],
) -> str:
    """Return one structured assistant response.

    Errors intentionally exclude response bodies because they may contain
    prompts, samples, or model-authored sensitive text.

    Args:
        messages: Chat messages to send.
        response_model: Pydantic model whose strict JSON schema constrains the response.

    Returns:
        The assistant message content.

    Raises:
        ParameterError: On authentication, authorization, or other permanent request rejection.
        TransientInferenceError: On network failures, timeouts, or retryable HTTP statuses.
        InvalidInferenceResponse: If the response envelope is malformed.
    """
    headers = {"Content-Type": "application/json"}
    if self._settings.api_key is not None:
        headers["Authorization"] = f"Bearer {self._settings.api_key}"
    payload = {
        "model": self._settings.model_id,
        "messages": list(messages),
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": response_model.__name__,
                "strict": True,
                "schema": _strict_json_schema(response_model),
            },
        },
        **self._settings.request_options,
    }
    try:
        response = httpx.post(
            self._settings.endpoint_url + _CHAT_COMPLETIONS_PATH,
            headers=headers,
            json=payload,
            timeout=self._settings.timeout_seconds,
            trust_env=self._trust_env,
        )
    except httpx.HTTPError as exc:
        raise TransientInferenceError("PII inference transport failed") from exc

    if response.status_code in {401, 403}:
        raise ParameterError(
            f"PII inference authentication or authorization failed (HTTP {response.status_code}); "
            "check NSS_INFERENCE_KEY or --inference-api-key"
        )
    if response.status_code in _TRANSIENT_STATUS_CODES or response.status_code >= 500:
        raise TransientInferenceError(
            f"PII inference service returned HTTP {response.status_code}",
            retry_after=_retry_after_seconds(response),
        )
    if response.status_code >= 400:
        raise ParameterError(f"PII inference request was rejected (HTTP {response.status_code})")

    try:
        envelope = response.json()
        content = envelope["choices"][0]["message"]["content"]
    except (ValueError, KeyError, IndexError, TypeError) as exc:
        raise InvalidInferenceResponse("PII inference service returned an invalid response envelope") from exc
    if not isinstance(content, str):
        raise InvalidInferenceResponse("PII inference service returned non-text response content")
    return content

nonblank(value)

Return value stripped of surrounding whitespace, or None if blank.

Source code in src/nemo_safe_synthesizer/config/inference_environment.py
def nonblank(value: str | None) -> str | None:
    """Return ``value`` stripped of surrounding whitespace, or ``None`` if blank."""
    if value is None:
        return None
    stripped = value.strip()
    return stripped or None

is_loopback_host(hostname)

Return whether hostname is localhost or a loopback IP address.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def is_loopback_host(hostname: str | None) -> bool:
    """Return whether ``hostname`` is ``localhost`` or a loopback IP address."""
    if hostname is None:
        return False
    if hostname.lower() == "localhost":
        return True
    try:
        return ipaddress.ip_address(hostname).is_loopback
    except ValueError:
        return False

merge_request_options(*layers)

Merge request-option layers field by field, dropping fields set to None.

Later layers win, and a None (JSON null) value removes the field, leaving it to the server's default.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def merge_request_options(*layers: Mapping[str, object]) -> dict[str, object]:
    """Merge request-option layers field by field, dropping fields set to ``None``.

    Later layers win, and a ``None`` (JSON ``null``) value removes the field,
    leaving it to the server's default.
    """
    merged: dict[str, object] = {}
    for layer in layers:
        merged.update(layer)
    return {field: value for field, value in merged.items() if value is not None}

reserved_request_fields(fields)

Return the sorted fields in fields that NSS sets itself.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def reserved_request_fields(fields: Iterable[str]) -> list[str]:
    """Return the sorted fields in ``fields`` that NSS sets itself."""
    return sorted(RESERVED_REQUEST_FIELDS.intersection(fields))

configured_model_id(config, environment)

Return config.model_id, else NSS_INFERENCE_MODEL, or None when neither is set.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def configured_model_id(config: LLMConfig, environment: InferenceEnvironment) -> str | None:
    """Return ``config.model_id``, else ``NSS_INFERENCE_MODEL``, or ``None`` when neither is set."""
    return nonblank(config.model_id) or environment.model_id

resolve_inference_settings(config, *, environ=None)

Resolve settings for an explicit inference endpoint from configuration and environment.

There are no defaults: an endpoint and its model must both be set. The model resolves from config.model_id, then NSS_INFERENCE_MODEL. As with other overlapping YAML and environment settings, an explicit YAML value wins and the environment only supplies its default. The CLI applies an explicit --inference-model-id to config.model_id before this runs.

The endpoint and API key are deliberately absent from persisted configuration. They come only from NSS_INFERENCE_ENDPOINT and NSS_INFERENCE_KEY, which the CLI populates from its runtime options. The per-request timeout comes from NSS_INFERENCE_TIMEOUT (default 60 seconds) and extra request fields from NSS_INFERENCE_REQUEST_OPTIONS; NSS sends no sampling settings of its own, so fields left out use the server's defaults. Plan discovery reaches this through resolve_planning_inference, which selects the managed local server instead when no endpoint is set.

Parameters:

Name Type Description Default
config LLMConfig

Persisted LLM behavior from replace_pii.llm.

required
environ Mapping[str, str] | None

Environment mapping to read; defaults to os.environ.

None

Returns:

Type Description
InferenceSettings

Settings with a validated endpoint and resolved model.

Raises:

Type Description
InvalidRequestSettingsError

If the timeout or request options are invalid.

ParameterError

If no endpoint is set, or the endpoint is not an absolute HTTP(S) URL, embeds credentials, or uses plaintext HTTP for a non-loopback host.

MissingInferenceModelError

If the endpoint has no model ID.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def resolve_inference_settings(
    config: LLMConfig,
    *,
    environ: Mapping[str, str] | None = None,
) -> InferenceSettings:
    """Resolve settings for an explicit inference endpoint from configuration and environment.

    There are no defaults: an endpoint and its model must both be set. The
    model resolves from ``config.model_id``, then ``NSS_INFERENCE_MODEL``. As
    with other overlapping YAML and environment settings, an explicit YAML
    value wins and the environment only supplies its default. The CLI applies
    an explicit ``--inference-model-id`` to ``config.model_id`` before this runs.

    The endpoint and API key are deliberately absent from persisted
    configuration. They come only from ``NSS_INFERENCE_ENDPOINT`` and
    ``NSS_INFERENCE_KEY``, which the CLI populates from its runtime options.
    The per-request timeout comes from ``NSS_INFERENCE_TIMEOUT`` (default 60
    seconds) and extra request fields from ``NSS_INFERENCE_REQUEST_OPTIONS``;
    NSS sends no sampling settings of its own, so fields left out use the
    server's defaults. Plan discovery reaches this through
    ``resolve_planning_inference``, which selects the managed local server
    instead when no endpoint is set.

    Args:
        config: Persisted LLM behavior from ``replace_pii.llm``.
        environ: Environment mapping to read; defaults to ``os.environ``.

    Returns:
        Settings with a validated endpoint and resolved model.

    Raises:
        InvalidRequestSettingsError: If the timeout or request options are invalid.
        ParameterError: If no endpoint is set, or the endpoint is not an
            absolute HTTP(S) URL, embeds credentials, or uses plaintext HTTP
            for a non-loopback host.
        MissingInferenceModelError: If the endpoint has no model ID.
    """
    return endpoint_inference_settings(config, InferenceEnvironment.from_environ(environ))

endpoint_inference_settings(config, environment)

Return settings for the explicit endpoint in environment; see resolve_inference_settings.

Source code in src/nemo_safe_synthesizer/pii_replacer/llm_client.py
def endpoint_inference_settings(config: LLMConfig, environment: InferenceEnvironment) -> InferenceSettings:
    """Return settings for the explicit endpoint in ``environment``; see ``resolve_inference_settings``."""
    resolved_endpoint = environment.endpoint
    resolved_model = configured_model_id(config, environment)
    if resolved_endpoint is None:
        raise ParameterError("No PII inference endpoint is set; set NSS_INFERENCE_ENDPOINT or --inference-endpoint-url")
    _validate_endpoint(resolved_endpoint)
    if resolved_model is None:
        raise MissingInferenceModelError(
            "NSS_INFERENCE_ENDPOINT is set, so the model it serves must be set too: use "
            "replace_pii.llm.model_id, --inference-model-id, or NSS_INFERENCE_MODEL"
        )

    return InferenceSettings(
        endpoint_url=resolved_endpoint.rstrip("/"),
        model_id=resolved_model,
        api_key=environment.api_key,
        max_workers=config.max_workers,
        timeout_seconds=(
            DEFAULT_NSS_INFERENCE_TIMEOUT_SECONDS
            if environment.timeout_seconds is None
            else environment.timeout_seconds
        ),
        request_options=merge_request_options(environment.request_options),
    )