replace_pii
replace_pii
¶
Dataset-specific PII detection/replacement plan.
This is the declarative, column-oriented plan a user (or an upstream detector) provides to describe how to replace PII in a dataset.
Every label is an Entity with an EntityAction and whether it
can_condition other columns. User-facing plans name the entity on each
replace target and on read-only depends_on entries. A conditioner that is
also a replace target inherits the entity from that target.
Classes:
| Name | Description |
|---|---|
EntityType |
Closed vocabulary for discovery and plan |
EntityAction |
What the engine does to a column of this entity type. |
PatternSyntax |
Grammar used by |
Entity |
One label in the entity vocabulary. |
ConditioningColumn |
An existing column that conditions the replacement of another column. |
PiiColumnPlan |
Replacement spec for one named column. |
PiiSamplerBackend |
Source of synthetic values for names and related person-like fields. |
PiiSamplerConfig |
Settings for Nemotron-Personas or Faker-backed person sampling. |
PiiReplacementPlan |
Dataset-specific replacement plan: the part of |
LLMConfig |
Inference behavior for LLM-assisted PII plan discovery. |
FreeTextDetectionConfig |
GLiNER2 and deterministic regex settings for free-text PII detection. |
ReplacePiiConfig |
Top-level |
Functions:
| Name | Description |
|---|---|
is_columns_to_replace_type |
Whether |
can_condition |
Whether |
validate_pattern_eligibility |
Validate context-free eligibility of a classification or plan pattern. |
validate_dependency_relationship |
Validate one dependency edge against the entity relationship catalog. |
Attributes:
| Name | Type | Description |
|---|---|---|
DEFAULT_GLINER2_MODEL_REVISION |
Hugging Face commit of |
|
DataToSamplerValueMapping |
Dependency-column values mapped to values understood by one sampler. |
|
FREE_TEXT_DETECTION_ENTITY_TYPES |
tuple[EntityType, ...]
|
Entity types eligible for fresh detection and replacement inside free text. |
REGEX_DETECTION_ENTITY_TYPES |
tuple[EntityType, ...]
|
Entity types detected exclusively by structurally validated built-in regex rules. |
GLINER_DETECTION_ENTITY_TYPES |
tuple[EntityType, ...]
|
Entity types requested from GLiNER2 rather than the built-in regex detector. |
DEFAULT_GLINER2_MODEL_REVISION = 'c153999da5f4c509df4322b0c6a1baf3d2c284d7'
module-attribute
¶
Hugging Face commit of DEFAULT_GLINER2_MODEL_ID that the default thresholds were tuned on.
DataToSamplerValueMapping = dict[str, dict[str, list[str]]]
module-attribute
¶
Dependency-column values mapped to values understood by one sampler.
Keyed by depends_on column name, then by dataset value. All keys and values are compared ignoring case.
FREE_TEXT_DETECTION_ENTITY_TYPES = tuple((entity.entity_type) for entity in ENTITIES if entity.action is EntityAction.REPLACE)
module-attribute
¶
Entity types eligible for fresh detection and replacement inside free text.
REGEX_DETECTION_ENTITY_TYPES = (EntityType.EMAIL, EntityType.CREDIT_DEBIT_CARD, EntityType.IPV4, EntityType.IPV6)
module-attribute
¶
Entity types detected exclusively by structurally validated built-in regex rules.
GLINER_DETECTION_ENTITY_TYPES = tuple(entity_type for entity_type in FREE_TEXT_DETECTION_ENTITY_TYPES if entity_type not in REGEX_DETECTION_ENTITY_TYPES)
module-attribute
¶
Entity types requested from GLiNER2 rather than the built-in regex detector.
The checkpoint's specific full_name label is used for complete names; its
broader person label is intentionally not requested.
EntityType
¶
Bases: StrEnum
Closed vocabulary for discovery and plan entity_type fields.
FREE_TEXT marks columns whose accepted PII spans are detected by
GLiNER2 and applicable deterministic regex rules, then replaced without
changing the surrounding text.
EntityAction
¶
Bases: Enum
What the engine does to a column of this entity type.
Attributes:
| Name | Type | Description |
|---|---|---|
REPLACE |
Synthesize a new cell value. |
|
REPLACE_IN_TEXT |
Rewrite PII spans inside free-form text, leaving the surrounding text intact. |
|
IDENTIFY_ONLY |
Label the column but never write to it. |
REPLACE = auto()
class-attribute
instance-attribute
¶
Synthesize a new cell value.
REPLACE_IN_TEXT = auto()
class-attribute
instance-attribute
¶
Rewrite PII spans inside free-form text, leaving the surrounding text intact.
Unlike REPLACE, the cell is not itself a single entity value: only the
spans accepted from GLiNER2 and applicable deterministic regex rules are
rewritten. Existing structured values and mappings do not create spans.
IDENTIFY_ONLY = auto()
class-attribute
instance-attribute
¶
Label the column but never write to it.
Cannot appear on columns_to_replace; may still condition other columns
when can_condition is set.
PatternSyntax
¶
Bases: Enum
Grammar used by PiiColumnPlan.pattern for an entity type.
Attributes:
| Name | Type | Description |
|---|---|---|
STRFTIME |
Python |
|
CHARACTER_MASK |
One drawn character per token, e.g. |
|
NAME_PARTS |
Whole-value templates composed of literals and name-part placeholders. |
STRFTIME = auto()
class-attribute
instance-attribute
¶
Python strftime codes, e.g. %m/%d/%Y or %d.%m.%y.
CHARACTER_MASK = auto()
class-attribute
instance-attribute
¶
One drawn character per token, e.g. pmc-###### or CUST-10[01]###.
NAME_PARTS = auto()
class-attribute
instance-attribute
¶
Whole-value templates composed of literals and name-part placeholders.
Entity(entity_type, action, can_condition, description, pattern_syntax=None)
dataclass
¶
One label in the entity vocabulary.
Attributes:
| Name | Type | Description |
|---|---|---|
entity_type |
EntityType
|
Label this entry defines, as named by |
action |
EntityAction
|
Engine treatment of a column carrying this entity. |
can_condition |
bool
|
Whether this entity may appear as |
description |
str
|
Short, general definition shown to the LLM classifier, so it can tell this entity from similar-looking data. |
pattern_syntax |
PatternSyntax | None
|
Notation |
entity_type
instance-attribute
¶
Label this entry defines, as named by entity_type in user-facing plans.
action
instance-attribute
¶
Engine treatment of a column carrying this entity.
can_condition
instance-attribute
¶
Whether this entity may appear as entity_type on a depends_on entry.
The executor applies columns_to_replace in DAG order and reads the
current cell as the conditioner, so a conditioner that is itself a replace
target supplies its post-replacement value.
description
instance-attribute
¶
Short, general definition shown to the LLM classifier, so it can tell this entity from similar-looking data.
pattern_syntax = None
class-attribute
instance-attribute
¶
Notation pattern uses, or None when this entity allows no pattern.
ConditioningColumn
pydantic-model
¶
Bases: NSSBaseModel
An existing column that conditions the replacement of another column.
entity_type is required for read-only conditioners (gender,
ethnic_background, city, …) that are not listed under
columns_to_replace.
When the conditioner is itself a replace target (e.g. email depends on
first_name), entity_type may be omitted: the plan fills it in from that
column's entry, and an explicit value must match it. Downstream nodes see the
cell after upstream replacements.
Fields:
-
column_name(str) -
entity_type(EntityType | None)
Validators:
-
_require_can_condition_when_set
column_name
pydantic-field
¶
Existing dataframe column that supplies the conditioning value.
entity_type = None
pydantic-field
¶
Entity this conditioning column holds. Required for read-only conditioners. Optional when this column_name is also in columns_to_replace: it is filled in from that entry's entity_type, and an explicit value must match it.
PiiColumnPlan
pydantic-model
¶
Bases: NSSBaseModel
Replacement spec for one named column.
entity_type must have action REPLACE or REPLACE_IN_TEXT, not
IDENTIFY_ONLY.
Identify-only types (date, gender, city, …) are invalid here.
depends_on is allowed only for types in ALLOWED_DEPENDS_ON.
Fields:
-
column_name(str) -
entity_type(EntityType) -
pattern(str | None) -
depends_on(list[ConditioningColumn])
Validators:
-
_validate_entity_and_depends_on
column_name
pydantic-field
¶
Name of the dataframe column to replace or scan.
entity_type
pydantic-field
¶
Entity this column holds. Identify-only entity types are not allowed here.
pattern = None
pydantic-field
¶
Optional whole-value format using the grammar associated with this entity type. Only entity types that define a pattern syntax may set this. When provided, the whole column is replaced with the pattern if it covers at least 99% of non-null values (checked against the dataframe, not here).
depends_on
pydantic-field
¶
Columns that condition the replacement of this column. Only entity types in ALLOWED_DEPENDS_ON may set this. Can omit ConditioningColumn.entity_type when the conditioner is listed in columns_to_replace. Matrix checks for explicit entity_type run here; omitted types and exclusive-group checks are resolved on PiiReplacementPlan.
PiiSamplerBackend
¶
Bases: StrEnum
Source of synthetic values for names and related person-like fields.
Attributes:
| Name | Type | Description |
|---|---|---|
NEMOTRON_PERSONAS |
Draw from downloaded Nemotron-Personas locale assets. |
|
FAKER |
Draw from the Faker library; ignores |
PiiSamplerConfig
pydantic-model
¶
Bases: NSSBaseModel
Settings for Nemotron-Personas or Faker-backed person sampling.
Fields:
-
backend(PiiSamplerBackend) -
nemotron_personas_path(str | None)
backend = PiiSamplerBackend.NEMOTRON_PERSONAS
pydantic-field
¶
Person sampler backend: downloaded Nemotron-Personas assets or Faker.
resolved_nemotron_personas_path()
¶
Return nemotron_personas_path if set, else the environment or built-in default.
Source code in src/nemo_safe_synthesizer/config/replace_pii.py
PiiReplacementPlan
pydantic-model
¶
Bases: Parameters
Dataset-specific replacement plan: the part of replace_pii a person reviews.
data_to_sampler_value_mapping targets the values of replace_pii.sampler
in replace_pii.locale, so a plan is checked against the sampler of each
run that uses it. columns_to_replace is a flat list whose
cross-column relationships are depends_on edges (a DAG). Context-free
dependency and graph checks are enforced here; plan-vs-dataframe checks
live in pii_replacer.planning.validation.
Fields:
Validators:
-
_reject_removed_legacy_fields -
_validate_data_to_sampler_value_mapping -
_reject_duplicate_replace_columns -
_resolve_omitted_depends_on_types -
_reject_duplicate_depends_on_entity_types -
_reject_exclusive_depends_on_groups -
_reject_dependency_cycles
columns_to_replace
pydantic-field
¶
Columns to replace or (for free_text) scan for PII spans to rewrite.
data_to_sampler_value_mapping
pydantic-field
¶
Maps dataset values in depends_on columns to the values used by replace_pii.sampler. Unlike most NSS settings, all matching here ignores case: dataset values, mapping keys, and sampler values, so Female, female, and FEMALE are the same value. For example, {sex: {Woman: [female], Non-binary: [female, male]}} makes rows where sex is Woman sample a female persona and rows where sex is Non-binary sample either. Gender values are female and male for every sampler; Nemotron-Personas also accepts its locale's own words, such as femme in fr_FR. Ethnicity values need Nemotron-Personas in a locale with ethnicity data. A dataset value without an entry is looked up in the sampler as is; a value that has no entry and matches no sampler value is an error. When replace_pii.replacement_plan is 'auto_discovery', NSS generates this mapping for you.
replacement_order()
¶
Return columns_to_replace so every column comes after the columns it depends on.
Independent columns keep their listed order. Dependencies on columns that are not replaced do not affect the order.
Raises:
| Type | Description |
|---|---|
ParameterError
|
If the dependencies form a cycle; the message names the columns in it. |
Source code in src/nemo_safe_synthesizer/config/replace_pii.py
LLMConfig
pydantic-model
¶
Bases: NSSBaseModel
Inference behavior for LLM-assisted PII plan discovery.
Automatic discovery always uses the LLM. Without an endpoint, NSS runs the
model in a managed local vLLM server. An
OpenAI-compatible endpoint is supplied at runtime through
NSS_INFERENCE_ENDPOINT or --inference-endpoint-url rather than persisted
in NSS configuration.
Fields:
-
model_id(str | None) -
max_workers(int)
max_workers = 8
pydantic-field
¶
Maximum concurrent requests for LLM-assisted plan discovery. Must be at least 1.
FreeTextDetectionConfig
pydantic-model
¶
Bases: NSSBaseModel
GLiNER2 and deterministic regex settings for free-text PII detection.
Fields:
-
model_id(str) -
model_revision(str | None) -
entity_thresholds(dict[EntityType, _ConfidenceThreshold]) -
checked_national_id_threshold(_ConfidenceThreshold) -
name_exclusion_thresholds(dict[str, _ConfidenceThreshold | None]) -
find_and_replace(bool) -
batch_size(int) -
chunk_length(int) -
chunk_overlap(int)
Validators:
-
_drop_default_revision_for_other_models -
_fill_default_entity_thresholds→entity_thresholds -
_reject_non_gliner_entity_thresholds→entity_thresholds -
_merge_default_name_exclusion_thresholds→name_exclusion_thresholds -
_validate_name_exclusion_labels→name_exclusion_thresholds -
_validate_chunk_overlap
model_id = DEFAULT_GLINER2_MODEL_ID
pydantic-field
¶
GLiNER2 model identifier used to detect PII spans in free-text columns. Must be nonempty.
model_revision = DEFAULT_GLINER2_MODEL_REVISION
pydantic-field
¶
Hugging Face revision (commit, branch, or tag) of model_id to load. Defaults to the commit of the default model that the default thresholds were tuned on. When model_id is changed and model_revision is not set, the latest revision is loaded. Set null to load the latest revision.
entity_thresholds
pydantic-field
¶
Minimum GLiNER2 confidence score per entity type. Entity types left out use the defaults: 0.944 for first_name; 0.7 for middle_name; 0.614 for last_name; 0.963 for full_name; 0.992 for phone_number; 0.9998 for date_of_birth; 0.987 for street_address; 0.979 for national_id; 0.991 for api_key; and 0.946 for unique_identifier. Email, credit_debit_card, ipv4, and ipv6 are detected by regex and take no threshold.
checked_national_id_threshold = _DEFAULT_CHECKED_NATIONAL_ID_THRESHOLD
pydantic-field
¶
Minimum GLiNER2 confidence score for a detected national ID that python-stdnum accepts as a personal ID of replace_pii.locale's country, such as a US Social Security number for en_US. It replaces entity_thresholds.national_id for those values, since the check rather than the score tells them apart. Locales without a check, such as en_GB, use only the national_id threshold.
name_exclusion_thresholds
pydantic-field
¶
Extra GLiNER2 labels, with their minimum confidence, for terms often mistaken for names, such as medical conditions or family relationships. A detected name that overlaps one of these spans is not replaced, and find_and_replace does not replace text inside them. Labels you set are merged into the defaults; set a label to null to drop it, and it stays null in the configuration. Dropping every label turns exclusion off, which finds more real names at the cost of more false names in clinical text.
find_and_replace = True
pydantic-field
¶
Search free text for values already known in the same record or group and replace the occurrences the detectors missed. Detected spans reuse a known value's replacement either way; this setting only adds the search. Known values are original values of structured replacement columns, including the words of a plain full name, and values detected elsewhere in free text. Only whole words are matched, ignoring case. Values shorter than three characters, one-word values that are common English words, such as 'Will' or 'Brown', and missing-value placeholders such as 'Unknown' are not searched for, and text inside a name-exclusion span, such as a medical condition, is never replaced this way. Set false to replace only detected spans, for example to measure what the search adds. The search runs only when replace_pii.locale is en_US, because the common-word list is English.
batch_size = 8
pydantic-field
¶
Number of text chunks processed in one GLiNER2 inference batch. Must be positive.
chunk_length = 384
pydantic-field
¶
Maximum length of each GLiNER2 text chunk. Must be positive.
chunk_overlap = 128
pydantic-field
¶
Overlap between adjacent GLiNER2 text chunks. Must be nonnegative and smaller than chunk_length.
ReplacePiiConfig
pydantic-model
¶
Bases: Parameters
Top-level replace_pii config.
replacement_plan holds what a person reviews: the columns to replace
and the value mapping. It is "auto_discovery" (the default), an inline
plan, or the path of a YAML plan file, resolved relative to the
configuration file that names it. locale, seed, and sampler
configure generation; a plan's value mapping is checked against the
sampler and locale of each run. llm and free_text_detection
configure how NSS finds PII.
Auto-discovery classifies columns with the LLM configured by llm, which
defaults to the NSS inference defaults. An inline plan bypasses discovery,
so it makes no LLM request; that is the way to replace PII without an LLM.
The API key remains a runtime secret supplied through NSS_INFERENCE_KEY
or the corresponding CLI option; it is never stored in this model.
v2 fields globals and steps are rejected with a removal error.
Fields:
-
schema_version(Literal[3]) -
locale(str) -
seed(int | None) -
sampler(PiiSamplerConfig) -
replacement_plan(SkipValidation[PiiReplacementPlan] | str) -
llm(LLMConfig) -
free_text_detection(FreeTextDetectionConfig)
Validators:
-
_reject_removed_legacy_fields -
_validate_schema_version -
_reject_null_llm→llm -
_resolve_personas_path→sampler -
_resolve_replacement_plan→replacement_plan
schema_version = _IMPLICIT_REPLACE_PII_SCHEMA_VERSION
pydantic-field
¶
Version of this replace_pii configuration schema. Missing versions are treated as version 3; this release accepts only version 3.
locale = 'en_US'
pydantic-field
¶
Locale for generated names and addresses, for reading phone numbers written without a country code, and of the Nemotron-Personas dataset (datasets/{locale}.parquet).
seed = None
pydantic-field
¶
Seed for synthetic value generation. When unset, each run draws a random seed and saves it in the resolved configuration. Anyone who knows the seed can test guesses for original values in the replaced data, so keep a configuration that contains it as private as the input data.
sampler
pydantic-field
¶
Sampler that generates names and addresses: Nemotron-Personas or Faker. When sampler is not set, NSS uses Nemotron-Personas, or Faker with a warning when the Nemotron-Personas data for locale is missing and the plan maps no ethnicity values. A relative nemotron_personas_path is resolved against the directory of the configuration file.
llm
pydantic-field
¶
Inference behavior for LLM-assisted plan discovery; the defaults run a local vLLM server. The endpoint is configured at runtime through NSS_INFERENCE_ENDPOINT or --inference-endpoint-url. An inline plan makes no LLM request.
free_text_detection
pydantic-field
¶
GLiNER2 and deterministic built-in regex settings for detecting PII spans in free-text columns.
is_auto_discovery
property
¶
Whether replacements should be auto-discovered (no explicit plan).
inline_plan
property
¶
The inline plan, or None for auto-discovery.
with_plan(plan)
¶
with_sampler_backend(backend)
¶
Return a copy whose sampler uses backend, keeping its other settings.
Source code in src/nemo_safe_synthesizer/config/replace_pii.py
is_columns_to_replace_type(entity_type)
¶
Whether entity_type may appear on columns_to_replace.
can_condition(entity_type)
¶
validate_pattern_eligibility(entity_type, pattern)
¶
Validate context-free eligibility of a classification or plan pattern.
Source code in src/nemo_safe_synthesizer/config/replace_pii.py
validate_dependency_relationship(target_entity_type, source_entity_type, *, target_column)
¶
Validate one dependency edge against the entity relationship catalog.
A missing source type is deferred until PiiReplacementPlan can infer it
from the complete replacement graph; the target must still support
dependencies at this earlier validation seam.