Skip to content

server

server

Start, probe, and stop one managed loopback vLLM server.

Classes:

Name Description
LocalVllmServer

Own one loopback vLLM server process for the duration of a with block.

Functions:

Name Description
is_vllm_installed

Return whether vLLM can be imported, without importing it.

local_runtime_problem

Return why this machine cannot run a managed vLLM server, or None if it can.

local_runtime_problem_message

Explain a local_runtime_problem result and how to use a remote service instead.

build_serve_command

Return the argv that launches vllm serve for profile.

LocalVllmServer(profile, *, host=DEFAULT_HOST, port=None, environ=None, request_options=None)

Own one loopback vLLM server process for the duration of a with block.

Entering starts vllm serve in its own process group, forwards its output to the NSS log at debug level, and waits until the server lists the profile's model and answers one strict-JSON-schema request. Exiting, or any startup failure including KeyboardInterrupt, stops the whole process group: SIGTERM first, SIGKILL after the profile's shutdown timeout. The server also receives SIGTERM if NSS dies without running its cleanup.

Parameters:

Name Type Description Default
profile LocalVllmProfile

Model and engine settings.

required
host str

Loopback host to bind.

DEFAULT_HOST
port int | None

Port to bind, or None to choose a free port.

None
environ Mapping[str, str] | None

Base environment for the server process; defaults to os.environ. NSS_INFERENCE_* values are not passed on.

None
request_options Mapping[str, object] | None

Chat-completions fields for the readiness probe and inference_settings; defaults to the profile's request options.

None

Raises:

Type Description
ParameterError

On entry, if vLLM, a CUDA GPU, or enough GPU memory is unavailable, or the port is in use.

GenerationError

On entry, if the server exits, times out, or fails its readiness checks.

Methods:

Name Description
inference_settings

Return client settings for this server, available once it has started.

start

Launch the server and block until it is ready.

stop

Stop the server's process group; safe to call more than once.

Attributes:

Name Type Description
endpoint_url str

OpenAI-compatible base URL, available once the server has started.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def __init__(
    self,
    profile: LocalVllmProfile,
    *,
    host: str = DEFAULT_HOST,
    port: int | None = None,
    environ: Mapping[str, str] | None = None,
    request_options: Mapping[str, object] | None = None,
) -> None:
    self._profile = profile
    self._host = host
    self._requested_port = port
    self._base_environ: Mapping[str, str] = os.environ if environ is None else environ
    self._request_options = merge_request_options(
        profile.request_options if request_options is None else request_options
    )
    self._api_key = secrets.token_urlsafe(32)
    self._port: int | None = None
    self._process: subprocess.Popen[str] | None = None
    self._output_thread: threading.Thread | None = None
    self._output_tail: deque[str] = deque(maxlen=_OUTPUT_TAIL_LINES)
    self._stopped = False

endpoint_url property

OpenAI-compatible base URL, available once the server has started.

inference_settings(*, max_workers=1, timeout_seconds=None)

Return client settings for this server, available once it has started.

Parameters:

Name Type Description Default
max_workers int

Maximum concurrent requests.

1
timeout_seconds float | None

Per-request timeout; defaults to the profile's request_timeout_seconds.

None
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def inference_settings(self, *, max_workers: int = 1, timeout_seconds: float | None = None) -> InferenceSettings:
    """Return client settings for this server, available once it has started.

    Args:
        max_workers: Maximum concurrent requests.
        timeout_seconds: Per-request timeout; defaults to the profile's
            ``request_timeout_seconds``.
    """
    return InferenceSettings(
        endpoint_url=self.endpoint_url,
        model_id=self._profile.served_name,
        max_workers=max_workers,
        api_key=self._api_key,
        timeout_seconds=self._profile.request_timeout_seconds if timeout_seconds is None else timeout_seconds,
        request_options=self._request_options,
    )

start()

Launch the server and block until it is ready.

Raises:

Type Description
ParameterError

If vLLM, a CUDA GPU, or enough GPU memory is unavailable, or the port is in use.

GenerationError

If the server exits, times out, or fails readiness checks.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def start(self) -> None:
    """Launch the server and block until it is ready.

    Raises:
        ParameterError: If vLLM, a CUDA GPU, or enough GPU memory is
            unavailable, or the port is in use.
        GenerationError: If the server exits, times out, or fails readiness checks.
    """
    if self._process is not None or self._stopped:
        raise InternalError("A LocalVllmServer can only be started once")
    if problem := local_runtime_problem(self._profile):
        raise ParameterError(local_runtime_problem_message(problem))
    self._port = _available_port(self._host, self._requested_port)
    command = build_serve_command(self._profile, host=self._host, port=self._port, parent_pid=os.getpid())
    logger.user.info(f"Starting local vLLM server for {self._profile.served_name!r} at {self.endpoint_url}")
    started = time.monotonic()
    try:
        self._process = _launch(command, self._server_environ())
        if self._process.stdout is not None:
            self._output_thread = threading.Thread(
                target=self._forward_output,
                args=(self._process.stdout,),
                name="nss-local-vllm-output",
                daemon=True,
            )
            self._output_thread.start()
        with heartbeat("Local vLLM server startup", logger_name=__name__, model=self._profile.served_name):
            self._wait_until_ready()
    except BaseException:
        self.stop()
        raise
    logger.user.info(
        f"Local vLLM server ready after {time.monotonic() - started:.0f}s",
        extra={"model": self._profile.served_name, "endpoint": self.endpoint_url},
    )

stop()

Stop the server's process group; safe to call more than once.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def stop(self) -> None:
    """Stop the server's process group; safe to call more than once."""
    if self._stopped:
        return
    self._stopped = True
    process = self._process
    if process is None:
        return
    # start_new_session makes the server its own process group leader.
    process_group = process.pid
    if process.poll() is None or _group_alive(process_group):
        _signal_group(process_group, signal.SIGTERM)
        try:
            stopped = self._wait_for_group_exit(process, process_group, self._profile.shutdown_timeout_seconds)
        except BaseException:
            # A second Ctrl-C during the graceful wait must not leave the server holding the GPU.
            _signal_group(process_group, signal.SIGKILL)
            raise
        if not stopped:
            logger.user.warning(
                f"Local vLLM server did not stop within {self._profile.shutdown_timeout_seconds:g}s; killing it"
            )
            _signal_group(process_group, signal.SIGKILL)
            if not self._wait_for_group_exit(process, process_group, _KILL_WAIT_SECONDS):
                logger.user.warning(f"Local vLLM server process group {process_group} is still running")
    if self._output_thread is not None:
        self._output_thread.join(timeout=_OUTPUT_JOIN_SECONDS)
    logger.user.info("Local vLLM server stopped")

is_vllm_installed()

Return whether vLLM can be imported, without importing it.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def is_vllm_installed() -> bool:
    """Return whether vLLM can be imported, without importing it."""
    return importlib.util.find_spec("vllm") is not None

local_runtime_problem(profile=None)

Return why this machine cannot run a managed vLLM server, or None if it can.

Parameters:

Name Type Description Default
profile LocalVllmProfile | None

When given, also check that enough GPUs are visible for its tensor_parallel_size and that each has its min_gpu_memory_gib.

None
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def local_runtime_problem(profile: LocalVllmProfile | None = None) -> str | None:
    """Return why this machine cannot run a managed vLLM server, or ``None`` if it can.

    Args:
        profile: When given, also check that enough GPUs are visible for its
            ``tensor_parallel_size`` and that each has its ``min_gpu_memory_gib``.
    """
    if not is_vllm_installed():
        return "vLLM is not installed (install the engine extra, for example `uv sync --extra cu129 --extra engine`)"
    import torch

    if not torch.cuda.is_available():
        return "no CUDA GPU is available"
    if profile is None:
        return None
    needed = profile.tensor_parallel_size
    visible = torch.cuda.device_count()
    if visible < needed:
        return (
            f"{profile.served_name!r} needs {needed} GPUs, but only {visible} {'is' if visible == 1 else 'are'} visible"
        )
    if profile.min_gpu_memory_gib is not None:
        # vLLM places tensor-parallel ranks on the first visible GPUs.
        for index in range(needed):
            properties = torch.cuda.get_device_properties(index)
            total_gib = properties.total_memory / 1024**3
            if total_gib < profile.min_gpu_memory_gib:
                return (
                    f"{profile.served_name!r} needs a GPU with at least {profile.min_gpu_memory_gib:g} GiB, "
                    f"but GPU {index} ({properties.name}) has {total_gib:.1f} GiB"
                )
    return None

local_runtime_problem_message(problem)

Explain a local_runtime_problem result and how to use a remote service instead.

Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def local_runtime_problem_message(problem: str) -> str:
    """Explain a ``local_runtime_problem`` result and how to use a remote service instead."""
    return (
        f"LLM-assisted PII planning runs a local vLLM server unless {ENDPOINT_ENV} is set, but {problem}. "
        f"Set {ENDPOINT_ENV}, plus {KEY_ENV} if it requires one, to use a remote OpenAI-compatible service instead. "
        "To replace PII without an LLM, give an explicit replacement plan; to skip replacement, pass --no-replace-pii "
        "or set replace_pii to null."
    )

build_serve_command(profile, *, host, port, parent_pid)

Return the argv that launches vllm serve for profile.

The command runs vLLM with the current interpreter, so it uses the same environment as NSS. The API key is passed separately through VLLM_API_KEY so it never appears in the process list.

Parameters:

Name Type Description Default
profile LocalVllmProfile

Model and engine settings.

required
host str

Loopback host to bind.

required
port int

Port to bind.

required
parent_pid int

PID the child checks before exec; the server stops when this process dies.

required
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
def build_serve_command(profile: LocalVllmProfile, *, host: str, port: int, parent_pid: int) -> list[str]:
    """Return the argv that launches ``vllm serve`` for ``profile``.

    The command runs vLLM with the current interpreter, so it uses the same
    environment as NSS. The API key is passed separately through
    ``VLLM_API_KEY`` so it never appears in the process list.

    Args:
        profile: Model and engine settings.
        host: Loopback host to bind.
        port: Port to bind.
        parent_pid: PID the child checks before exec; the server stops when
            this process dies.
    """
    options = [
        "--revision",
        profile.revision,
        "--served-model-name",
        profile.served_name,
        "--host",
        host,
        "--port",
        str(port),
        "--gpu-memory-utilization",
        str(profile.gpu_memory_utilization),
        "--tensor-parallel-size",
        str(profile.tensor_parallel_size),
    ]
    if profile.max_model_len is not None:
        options += ["--max-model-len", str(profile.max_model_len)]
    if profile.max_num_seqs is not None:
        options += ["--max-num-seqs", str(profile.max_num_seqs)]
    return [
        sys.executable,
        "-c",
        _PARENT_DEATH_LAUNCHER,
        str(parent_pid),
        "-m",
        "vllm.entrypoints.cli.main",
        "serve",
        profile.model_id,
        *options,
        *profile.extra_args,
    ]