profile
profile
¶
Pinned vLLM serving profiles for the managed local PII planning server.
Classes:
| Name | Description |
|---|---|
LocalVllmProfile |
Pinned model and vLLM engine settings for one locally served planner model. |
Functions:
| Name | Description |
|---|---|
bundled_profile_names |
Return the names of profiles shipped with NSS. |
bundled_profile_for_model |
Return the bundled profile whose served model name is |
load_profile |
Load a bundled profile by name, or a profile YAML file by path. |
Attributes:
| Name | Type | Description |
|---|---|---|
RESERVED_VLLM_OPTIONS |
|
RESERVED_VLLM_OPTIONS = frozenset({'--api-key', '--config', '--enable-log-outputs', '--enable-log-requests', '--host', '--model', '--port', '--revision', '--served-model-name', '--tokenizer-revision', '--uds'})
module-attribute
¶
vllm serve options NSS sets itself or keeps disabled.
The launch owns the address and credential, the profile owns the model
identity, and request logging stays off because planning prompts contain raw
cell samples. --config is reserved because vLLM reads any option, including
these, from the config file it names.
LocalVllmProfile
pydantic-model
¶
Bases: BaseModel
Pinned model and vLLM engine settings for one locally served planner model.
Profiles describe only the model and engine. The listening address comes
from NSS_INFERENCE_ENDPOINT or a free loopback port, and every launch
generates its own API key.
Config:
extra:forbidfrozen:True
Fields:
-
model_id(str) -
revision(str) -
served_model_name(str | None) -
gpu_memory_utilization(float) -
max_model_len(int | None) -
max_num_seqs(int | None) -
tensor_parallel_size(int) -
min_gpu_memory_gib(float | None) -
extra_args(tuple[str, ...]) -
request_options(dict[str, JsonValue]) -
environment(dict[str, str]) -
request_timeout_seconds(float) -
startup_timeout_seconds(float) -
shutdown_timeout_seconds(float)
Validators:
-
_reject_reserved_request_fields→request_options -
_reject_managed_environment→environment -
_reject_reserved_options→extra_args
model_id
pydantic-field
¶
Hugging Face model repository or local model directory.
revision
pydantic-field
¶
Pinned model revision, preferably a full commit SHA.
served_model_name = None
pydantic-field
¶
Model name exposed by the server. Defaults to model_id.
gpu_memory_utilization = 0.9
pydantic-field
¶
Fraction of GPU memory vLLM may reserve. Must be in (0, 1].
max_model_len = None
pydantic-field
¶
Maximum prompt plus completion tokens per request. Uses the model's limit when unset.
max_num_seqs = None
pydantic-field
¶
Maximum concurrently scheduled requests. Uses the vLLM default when unset.
tensor_parallel_size = 1
pydantic-field
¶
Number of GPUs used for tensor parallelism.
min_gpu_memory_gib = None
pydantic-field
¶
Total memory, in GiB, each of the tensor_parallel_size GPUs must have. NSS checks it before downloading weights or launching the server; unset skips the check.
extra_args = ()
pydantic-field
¶
Additional vllm serve options. Options NSS manages, such as --host, --port, --api-key, and --enable-log-requests, are rejected.
request_options
pydantic-field
¶
Chat-completions fields sent with every planning request, such as temperature, top_p, top_k, and thinking_token_budget. Fields left out use the server's defaults. NSS_INFERENCE_REQUEST_OPTIONS overrides individual fields; a null value drops a field.
environment
pydantic-field
¶
Extra environment variables for the vLLM server process, such as VLLM_USE_V2_MODEL_RUNNER=0, which the bundled profiles set so thinking_token_budget takes effect.
request_timeout_seconds = DEFAULT_NSS_INFERENCE_TIMEOUT_SECONDS
pydantic-field
¶
Per-request timeout for planning calls to this model, including reasoning. NSS_INFERENCE_TIMEOUT overrides it.
startup_timeout_seconds = 600
pydantic-field
¶
Maximum seconds to wait for the server to become ready, including any model download.
shutdown_timeout_seconds = 30
pydantic-field
¶
Seconds to wait after SIGTERM before force-killing the server.
served_name
property
¶
Model name clients must send; served_model_name or model_id.
bundled_profile_names()
¶
Return the names of profiles shipped with NSS.
bundled_profile_for_model(model_id)
¶
Return the bundled profile whose served model name is model_id, if any.
Source code in src/nemo_safe_synthesizer/pii_replacer/local_inference/profile.py
load_profile(reference)
¶
Load a bundled profile by name, or a profile YAML file by path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reference
|
str | Path
|
Bundled profile name, such as |
required |
Returns:
| Type | Description |
|---|---|
LocalVllmProfile
|
The validated profile. |
Raises:
| Type | Description |
|---|---|
ParameterError
|
If the reference matches neither a bundled profile nor a file, or the file is not a valid profile. |