server
server
¶
Start, probe, and stop one managed loopback vLLM server.
Classes:
| Name | Description |
|---|---|
LocalVllmServer |
Own one loopback vLLM server process for the duration of a |
Functions:
| Name | Description |
|---|---|
is_vllm_installed |
Return whether vLLM can be imported, without importing it. |
local_runtime_problem |
Return why this machine cannot run a managed vLLM server, or |
local_runtime_problem_message |
Explain a |
build_serve_command |
Return the argv that launches |
LocalVllmServer(profile, *, host=DEFAULT_HOST, port=None, environ=None, request_options=None)
¶
Own one loopback vLLM server process for the duration of a with block.
Entering starts vllm serve in its own process group, forwards its
output to the NSS log at debug level, and waits until the server lists the
profile's model and answers one strict-JSON-schema request. Exiting, or
any startup failure including KeyboardInterrupt, stops the whole
process group: SIGTERM first, SIGKILL after the profile's shutdown timeout.
The server also receives SIGTERM if NSS dies without running its cleanup.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
LocalVllmProfile
|
Model and engine settings. |
required |
host
|
str
|
Loopback host to bind. |
DEFAULT_HOST
|
port
|
int | None
|
Port to bind, or |
None
|
environ
|
Mapping[str, str] | None
|
Base environment for the server process; defaults to
|
None
|
request_options
|
Mapping[str, object] | None
|
Chat-completions fields for the readiness probe and
|
None
|
Raises:
| Type | Description |
|---|---|
ParameterError
|
On entry, if vLLM, a CUDA GPU, or enough GPU memory is unavailable, or the port is in use. |
GenerationError
|
On entry, if the server exits, times out, or fails its readiness checks. |
Methods:
| Name | Description |
|---|---|
inference_settings |
Return client settings for this server, available once it has started. |
start |
Launch the server and block until it is ready. |
stop |
Stop the server's process group; safe to call more than once. |
Attributes:
| Name | Type | Description |
|---|---|---|
endpoint_url |
str
|
OpenAI-compatible base URL, available once the server has started. |
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
endpoint_url
property
¶
OpenAI-compatible base URL, available once the server has started.
inference_settings(*, max_workers=1, timeout_seconds=None)
¶
Return client settings for this server, available once it has started.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
max_workers
|
int
|
Maximum concurrent requests. |
1
|
timeout_seconds
|
float | None
|
Per-request timeout; defaults to the profile's
|
None
|
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
start()
¶
Launch the server and block until it is ready.
Raises:
| Type | Description |
|---|---|
ParameterError
|
If vLLM, a CUDA GPU, or enough GPU memory is unavailable, or the port is in use. |
GenerationError
|
If the server exits, times out, or fails readiness checks. |
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
stop()
¶
Stop the server's process group; safe to call more than once.
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
is_vllm_installed()
¶
local_runtime_problem(profile=None)
¶
Return why this machine cannot run a managed vLLM server, or None if it can.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
LocalVllmProfile | None
|
When given, also check that enough GPUs are visible for its
|
None
|
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
local_runtime_problem_message(problem)
¶
Explain a local_runtime_problem result and how to use a remote service instead.
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/server.py
build_serve_command(profile, *, host, port, parent_pid)
¶
Return the argv that launches vllm serve for profile.
The command runs vLLM with the current interpreter, so it uses the same
environment as NSS. The API key is passed separately through
VLLM_API_KEY so it never appears in the process list.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
LocalVllmProfile
|
Model and engine settings. |
required |
host
|
str
|
Loopback host to bind. |
required |
port
|
int
|
Port to bind. |
required |
parent_pid
|
int
|
PID the child checks before exec; the server stops when this process dies. |
required |