Skip to content

replace_pii

replace_pii

Dataset-specific PII detection/replacement plan.

This is the declarative, column-oriented plan a user (or an upstream detector) provides to describe how to replace PII in a dataset.

Every label is an Entity with an EntityAction and whether it can_condition other columns. User-facing plans name the entity on each replace target and on read-only depends_on entries. A conditioner that is also a replace target inherits the entity from that target.

Classes:

Name Description
EntityType

Closed vocabulary for discovery and plan entity_type fields.

EntityAction

What the engine does to a column of this entity type.

PatternSyntax

Grammar used by PiiColumnPlan.pattern for an entity type.

Entity

One label in the entity vocabulary.

ConditioningColumn

An existing column that conditions the replacement of another column.

PiiColumnPlan

Replacement spec for one named column.

PiiSamplerBackend

Source of synthetic values for names and related person-like fields.

PiiSamplerConfig

Settings for Nemotron-Personas or Faker-backed person sampling.

PiiReplacementPlan

Dataset-specific replacement plan: the part of replace_pii a person reviews.

LLMConfig

Inference behavior for LLM-assisted PII plan discovery.

FreeTextDetectionConfig

GLiNER2 and deterministic regex settings for free-text PII detection.

ReplacePiiConfig

Top-level replace_pii config.

Functions:

Name Description
is_columns_to_replace_type

Whether entity_type may appear on columns_to_replace.

can_condition

Whether entity_type may appear on a depends_on entry.

validate_pattern_eligibility

Validate context-free eligibility of a classification or plan pattern.

validate_dependency_relationship

Validate one dependency edge against the entity relationship catalog.

Attributes:

Name Type Description
DEFAULT_GLINER2_MODEL_REVISION

Hugging Face commit of DEFAULT_GLINER2_MODEL_ID that the default thresholds were tuned on.

DataToSamplerValueMapping

Dependency-column values mapped to values understood by one sampler.

FREE_TEXT_DETECTION_ENTITY_TYPES tuple[EntityType, ...]

Entity types eligible for fresh detection and replacement inside free text.

REGEX_DETECTION_ENTITY_TYPES tuple[EntityType, ...]

Entity types detected exclusively by structurally validated built-in regex rules.

GLINER_DETECTION_ENTITY_TYPES tuple[EntityType, ...]

Entity types requested from GLiNER2 rather than the built-in regex detector.

DEFAULT_GLINER2_MODEL_REVISION = 'c153999da5f4c509df4322b0c6a1baf3d2c284d7' module-attribute

Hugging Face commit of DEFAULT_GLINER2_MODEL_ID that the default thresholds were tuned on.

DataToSamplerValueMapping = dict[str, dict[str, list[str]]] module-attribute

Dependency-column values mapped to values understood by one sampler.

Keyed by depends_on column name, then by dataset value. All keys and values are compared ignoring case.

FREE_TEXT_DETECTION_ENTITY_TYPES = tuple((entity.entity_type) for entity in ENTITIES if entity.action is EntityAction.REPLACE) module-attribute

Entity types eligible for fresh detection and replacement inside free text.

REGEX_DETECTION_ENTITY_TYPES = (EntityType.EMAIL, EntityType.CREDIT_DEBIT_CARD, EntityType.IPV4, EntityType.IPV6) module-attribute

Entity types detected exclusively by structurally validated built-in regex rules.

GLINER_DETECTION_ENTITY_TYPES = tuple(entity_type for entity_type in FREE_TEXT_DETECTION_ENTITY_TYPES if entity_type not in REGEX_DETECTION_ENTITY_TYPES) module-attribute

Entity types requested from GLiNER2 rather than the built-in regex detector.

The checkpoint's specific full_name label is used for complete names; its broader person label is intentionally not requested.

EntityType

Bases: StrEnum

Closed vocabulary for discovery and plan entity_type fields.

FREE_TEXT marks columns whose accepted PII spans are detected by GLiNER2 and applicable deterministic regex rules, then replaced without changing the surrounding text.

EntityAction

Bases: Enum

What the engine does to a column of this entity type.

Attributes:

Name Type Description
REPLACE

Synthesize a new cell value.

REPLACE_IN_TEXT

Rewrite PII spans inside free-form text, leaving the surrounding text intact.

IDENTIFY_ONLY

Label the column but never write to it.

REPLACE = auto() class-attribute instance-attribute

Synthesize a new cell value.

REPLACE_IN_TEXT = auto() class-attribute instance-attribute

Rewrite PII spans inside free-form text, leaving the surrounding text intact.

Unlike REPLACE, the cell is not itself a single entity value: only the spans accepted from GLiNER2 and applicable deterministic regex rules are rewritten. Existing structured values and mappings do not create spans.

IDENTIFY_ONLY = auto() class-attribute instance-attribute

Label the column but never write to it.

Cannot appear on columns_to_replace; may still condition other columns when can_condition is set.

PatternSyntax

Bases: Enum

Grammar used by PiiColumnPlan.pattern for an entity type.

Attributes:

Name Type Description
STRFTIME

Python strftime codes, e.g. %m/%d/%Y or %d.%m.%y.

CHARACTER_MASK

One drawn character per token, e.g. pmc-###### or CUST-10[01]###.

NAME_PARTS

Whole-value templates composed of literals and name-part placeholders.

STRFTIME = auto() class-attribute instance-attribute

Python strftime codes, e.g. %m/%d/%Y or %d.%m.%y.

CHARACTER_MASK = auto() class-attribute instance-attribute

One drawn character per token, e.g. pmc-###### or CUST-10[01]###.

NAME_PARTS = auto() class-attribute instance-attribute

Whole-value templates composed of literals and name-part placeholders.

Entity(entity_type, action, can_condition, description, pattern_syntax=None) dataclass

One label in the entity vocabulary.

Attributes:

Name Type Description
entity_type EntityType

Label this entry defines, as named by entity_type in user-facing plans.

action EntityAction

Engine treatment of a column carrying this entity.

can_condition bool

Whether this entity may appear as entity_type on a depends_on entry.

description str

Short, general definition shown to the LLM classifier, so it can tell this entity from similar-looking data.

pattern_syntax PatternSyntax | None

Notation pattern uses, or None when this entity allows no pattern.

entity_type instance-attribute

Label this entry defines, as named by entity_type in user-facing plans.

action instance-attribute

Engine treatment of a column carrying this entity.

can_condition instance-attribute

Whether this entity may appear as entity_type on a depends_on entry.

The executor applies columns_to_replace in DAG order and reads the current cell as the conditioner, so a conditioner that is itself a replace target supplies its post-replacement value.

description instance-attribute

Short, general definition shown to the LLM classifier, so it can tell this entity from similar-looking data.

pattern_syntax = None class-attribute instance-attribute

Notation pattern uses, or None when this entity allows no pattern.

ConditioningColumn pydantic-model

Bases: NSSBaseModel

An existing column that conditions the replacement of another column.

entity_type is required for read-only conditioners (gender, ethnic_background, city, …) that are not listed under columns_to_replace.

When the conditioner is itself a replace target (e.g. email depends on first_name), entity_type may be omitted: the plan fills it in from that column's entry, and an explicit value must match it. Downstream nodes see the cell after upstream replacements.

Fields:

Validators:

  • _require_can_condition_when_set

column_name pydantic-field

Existing dataframe column that supplies the conditioning value.

entity_type = None pydantic-field

Entity this conditioning column holds. Required for read-only conditioners. Optional when this column_name is also in columns_to_replace: it is filled in from that entry's entity_type, and an explicit value must match it.

PiiColumnPlan pydantic-model

Bases: NSSBaseModel

Replacement spec for one named column.

entity_type must have action REPLACE or REPLACE_IN_TEXT, not IDENTIFY_ONLY. Identify-only types (date, gender, city, …) are invalid here. depends_on is allowed only for types in ALLOWED_DEPENDS_ON.

Fields:

Validators:

  • _validate_entity_and_depends_on

column_name pydantic-field

Name of the dataframe column to replace or scan.

entity_type pydantic-field

Entity this column holds. Identify-only entity types are not allowed here.

pattern = None pydantic-field

Optional whole-value format using the grammar associated with this entity type. Only entity types that define a pattern syntax may set this. When provided, the whole column is replaced with the pattern if it covers at least 99% of non-null values (checked against the dataframe, not here).

depends_on pydantic-field

Columns that condition the replacement of this column. Only entity types in ALLOWED_DEPENDS_ON may set this. Can omit ConditioningColumn.entity_type when the conditioner is listed in columns_to_replace. Matrix checks for explicit entity_type run here; omitted types and exclusive-group checks are resolved on PiiReplacementPlan.

PiiSamplerBackend

Bases: StrEnum

Source of synthetic values for names and related person-like fields.

Attributes:

Name Type Description
NEMOTRON_PERSONAS

Draw from downloaded Nemotron-Personas locale assets.

FAKER

Draw from the Faker library; ignores ethnic_background conditioners.

NEMOTRON_PERSONAS = 'nemotron-personas' class-attribute instance-attribute

Draw from downloaded Nemotron-Personas locale assets.

FAKER = 'faker' class-attribute instance-attribute

Draw from the Faker library; ignores ethnic_background conditioners.

PiiSamplerConfig pydantic-model

Bases: NSSBaseModel

Settings for Nemotron-Personas or Faker-backed person sampling.

Fields:

backend = PiiSamplerBackend.NEMOTRON_PERSONAS pydantic-field

Person sampler backend: downloaded Nemotron-Personas assets or Faker.

resolved_nemotron_personas_path()

Return nemotron_personas_path if set, else the environment or built-in default.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def resolved_nemotron_personas_path(self) -> Path:
    """Return ``nemotron_personas_path`` if set, else the environment or built-in default."""
    if self.nemotron_personas_path is not None:
        return Path(self.nemotron_personas_path).expanduser()
    return default_nemotron_personas_path()

PiiReplacementPlan pydantic-model

Bases: Parameters

Dataset-specific replacement plan: the part of replace_pii a person reviews.

data_to_sampler_value_mapping targets the values of replace_pii.sampler in replace_pii.locale, so a plan is checked against the sampler of each run that uses it. columns_to_replace is a flat list whose cross-column relationships are depends_on edges (a DAG). Context-free dependency and graph checks are enforced here; plan-vs-dataframe checks live in pii_replacer.planning.validation.

Fields:

Validators:

  • _reject_removed_legacy_fields
  • _validate_data_to_sampler_value_mapping
  • _reject_duplicate_replace_columns
  • _resolve_omitted_depends_on_types
  • _reject_duplicate_depends_on_entity_types
  • _reject_exclusive_depends_on_groups
  • _reject_dependency_cycles

columns_to_replace pydantic-field

Columns to replace or (for free_text) scan for PII spans to rewrite.

data_to_sampler_value_mapping pydantic-field

Maps dataset values in depends_on columns to the values used by replace_pii.sampler. Unlike most NSS settings, all matching here ignores case: dataset values, mapping keys, and sampler values, so Female, female, and FEMALE are the same value. For example, {sex: {Woman: [female], Non-binary: [female, male]}} makes rows where sex is Woman sample a female persona and rows where sex is Non-binary sample either. Gender values are female and male for every sampler; Nemotron-Personas also accepts its locale's own words, such as femme in fr_FR. Ethnicity values need Nemotron-Personas in a locale with ethnicity data. A dataset value without an entry is looked up in the sampler as is; a value that has no entry and matches no sampler value is an error. When replace_pii.replacement_plan is 'auto_discovery', NSS generates this mapping for you.

replacement_order()

Return columns_to_replace so every column comes after the columns it depends on.

Independent columns keep their listed order. Dependencies on columns that are not replaced do not affect the order.

Raises:

Type Description
ParameterError

If the dependencies form a cycle; the message names the columns in it.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def replacement_order(self) -> tuple[PiiColumnPlan, ...]:
    """Return ``columns_to_replace`` so every column comes after the columns it depends on.

    Independent columns keep their listed order. Dependencies on columns that are not replaced
    do not affect the order.

    Raises:
        ParameterError: If the dependencies form a cycle; the message names the columns in it.
    """
    specs = self.columns_to_replace
    by_name = {spec.column_name: spec for spec in specs}
    position = {spec.column_name: index for index, spec in enumerate(specs)}
    outgoing: dict[str, set[str]] = {column: set() for column in by_name}
    indegree = dict.fromkeys(by_name, 0)
    for spec in specs:
        for dependency in spec.depends_on:
            source = dependency.column_name
            if source not in by_name or spec.column_name in outgoing[source]:
                continue
            outgoing[source].add(spec.column_name)
            indegree[spec.column_name] += 1

    ready = [(position[column], column) for column, degree in indegree.items() if degree == 0]
    heapq.heapify(ready)
    ordered: list[PiiColumnPlan] = []
    while ready:
        _, source = heapq.heappop(ready)
        ordered.append(by_name[source])
        for target in outgoing[source]:
            indegree[target] -= 1
            if indegree[target] == 0:
                heapq.heappush(ready, (position[target], target))

    if len(ordered) != len(by_name):
        cycle_columns = sorted(column for column, degree in indegree.items() if degree > 0)
        raise ParameterError(
            "replacement dependencies contain a cycle involving: "
            + ", ".join(repr(column) for column in cycle_columns)
        )
    return tuple(ordered)

LLMConfig pydantic-model

Bases: NSSBaseModel

Inference behavior for LLM-assisted PII plan discovery.

Automatic discovery always uses the LLM. Without an endpoint, NSS runs the model in a managed local vLLM server. An OpenAI-compatible endpoint is supplied at runtime through NSS_INFERENCE_ENDPOINT or --inference-endpoint-url rather than persisted in NSS configuration.

Fields:

max_workers = 8 pydantic-field

Maximum concurrent requests for LLM-assisted plan discovery. Must be at least 1.

FreeTextDetectionConfig pydantic-model

Bases: NSSBaseModel

GLiNER2 and deterministic regex settings for free-text PII detection.

Fields:

Validators:

model_id = DEFAULT_GLINER2_MODEL_ID pydantic-field

GLiNER2 model identifier used to detect PII spans in free-text columns. Must be nonempty.

model_revision = DEFAULT_GLINER2_MODEL_REVISION pydantic-field

Hugging Face revision (commit, branch, or tag) of model_id to load. Defaults to the commit of the default model that the default thresholds were tuned on. When model_id is changed and model_revision is not set, the latest revision is loaded. Set null to load the latest revision.

entity_thresholds pydantic-field

Minimum GLiNER2 confidence score per entity type. Entity types left out use the defaults: 0.944 for first_name; 0.7 for middle_name; 0.614 for last_name; 0.963 for full_name; 0.992 for phone_number; 0.9998 for date_of_birth; 0.987 for street_address; 0.979 for national_id; 0.991 for api_key; and 0.946 for unique_identifier. Email, credit_debit_card, ipv4, and ipv6 are detected by regex and take no threshold.

checked_national_id_threshold = _DEFAULT_CHECKED_NATIONAL_ID_THRESHOLD pydantic-field

Minimum GLiNER2 confidence score for a detected national ID that python-stdnum accepts as a personal ID of replace_pii.locale's country, such as a US Social Security number for en_US. It replaces entity_thresholds.national_id for those values, since the check rather than the score tells them apart. Locales without a check, such as en_GB, use only the national_id threshold.

name_exclusion_thresholds pydantic-field

Extra GLiNER2 labels, with their minimum confidence, for terms often mistaken for names, such as medical conditions or family relationships. A detected name that overlaps one of these spans is not replaced, and find_and_replace does not replace text inside them. Labels you set are merged into the defaults; set a label to null to drop it, and it stays null in the configuration. Dropping every label turns exclusion off, which finds more real names at the cost of more false names in clinical text.

find_and_replace = True pydantic-field

Search free text for values already known in the same record or group and replace the occurrences the detectors missed. Detected spans reuse a known value's replacement either way; this setting only adds the search. Known values are original values of structured replacement columns, including the words of a plain full name, and values detected elsewhere in free text. Only whole words are matched, ignoring case. Values shorter than three characters, one-word values that are common English words, such as 'Will' or 'Brown', and missing-value placeholders such as 'Unknown' are not searched for, and text inside a name-exclusion span, such as a medical condition, is never replaced this way. Set false to replace only detected spans, for example to measure what the search adds. The search runs only when replace_pii.locale is en_US, because the common-word list is English.

batch_size = 8 pydantic-field

Number of text chunks processed in one GLiNER2 inference batch. Must be positive.

chunk_length = 384 pydantic-field

Maximum length of each GLiNER2 text chunk. Must be positive.

chunk_overlap = 128 pydantic-field

Overlap between adjacent GLiNER2 text chunks. Must be nonnegative and smaller than chunk_length.

ReplacePiiConfig pydantic-model

Bases: Parameters

Top-level replace_pii config.

replacement_plan holds what a person reviews: the columns to replace and the value mapping. It is "auto_discovery" (the default), an inline plan, or the path of a YAML plan file, resolved relative to the configuration file that names it. locale, seed, and sampler configure generation; a plan's value mapping is checked against the sampler and locale of each run. llm and free_text_detection configure how NSS finds PII.

Auto-discovery classifies columns with the LLM configured by llm, which defaults to the NSS inference defaults. An inline plan bypasses discovery, so it makes no LLM request; that is the way to replace PII without an LLM. The API key remains a runtime secret supplied through NSS_INFERENCE_KEY or the corresponding CLI option; it is never stored in this model.

v2 fields globals and steps are rejected with a removal error.

Fields:

Validators:

  • _reject_removed_legacy_fields
  • _validate_schema_version
  • _reject_null_llm → llm
  • _resolve_personas_path → sampler
  • _resolve_replacement_plan → replacement_plan

schema_version = _IMPLICIT_REPLACE_PII_SCHEMA_VERSION pydantic-field

Version of this replace_pii configuration schema. Missing versions are treated as version 3; this release accepts only version 3.

locale = 'en_US' pydantic-field

Locale for generated names and addresses, for reading phone numbers written without a country code, and of the Nemotron-Personas dataset (datasets/{locale}.parquet).

seed = None pydantic-field

Seed for synthetic value generation. When unset, each run draws a random seed and saves it in the resolved configuration. Anyone who knows the seed can test guesses for original values in the replaced data, so keep a configuration that contains it as private as the input data.

sampler pydantic-field

Sampler that generates names and addresses: Nemotron-Personas or Faker. When sampler is not set, NSS uses Nemotron-Personas, or Faker with a warning when the Nemotron-Personas data for locale is missing and the plan maps no ethnicity values. A relative nemotron_personas_path is resolved against the directory of the configuration file.

llm pydantic-field

Inference behavior for LLM-assisted plan discovery; the defaults run a local vLLM server. The endpoint is configured at runtime through NSS_INFERENCE_ENDPOINT or --inference-endpoint-url. An inline plan makes no LLM request.

free_text_detection pydantic-field

GLiNER2 and deterministic built-in regex settings for detecting PII spans in free-text columns.

is_auto_discovery property

Whether replacements should be auto-discovered (no explicit plan).

inline_plan property

The inline plan, or None for auto-discovery.

with_plan(plan)

Return a copy whose replacement_plan is plan.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def with_plan(self, plan: PiiReplacementPlan) -> Self:
    """Return a copy whose ``replacement_plan`` is ``plan``."""
    return self.model_copy(update={"replacement_plan": plan})

with_sampler_backend(backend)

Return a copy whose sampler uses backend, keeping its other settings.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def with_sampler_backend(self, backend: PiiSamplerBackend) -> Self:
    """Return a copy whose ``sampler`` uses ``backend``, keeping its other settings."""
    return self.model_copy(update={"sampler": self.sampler.model_copy(update={"backend": backend})})

is_columns_to_replace_type(entity_type)

Whether entity_type may appear on columns_to_replace.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def is_columns_to_replace_type(entity_type: EntityType) -> bool:
    """Whether ``entity_type`` may appear on ``columns_to_replace``."""
    return ENTITY_BY_TYPE[entity_type].action is not EntityAction.IDENTIFY_ONLY

can_condition(entity_type)

Whether entity_type may appear on a depends_on entry.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def can_condition(entity_type: EntityType) -> bool:
    """Whether ``entity_type`` may appear on a ``depends_on`` entry."""
    return ENTITY_BY_TYPE[entity_type].can_condition

validate_pattern_eligibility(entity_type, pattern)

Validate context-free eligibility of a classification or plan pattern.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def validate_pattern_eligibility(entity_type: EntityType | None, pattern: str | None) -> None:
    """Validate context-free eligibility of a classification or plan pattern."""
    if pattern is None:
        return
    if not pattern.strip():
        raise ParameterError("pattern must be non-empty when provided")
    if entity_type is None:
        raise ParameterError("unclassified columns cannot include a pattern")
    if ENTITY_BY_TYPE[entity_type].pattern_syntax is None:
        raise ParameterError(f"entity_type {entity_type.value!r} does not allow pattern")

validate_dependency_relationship(target_entity_type, source_entity_type, *, target_column)

Validate one dependency edge against the entity relationship catalog.

A missing source type is deferred until PiiReplacementPlan can infer it from the complete replacement graph; the target must still support dependencies at this earlier validation seam.

Source code in src/nemo_safe_synthesizer/config/replace_pii.py
def validate_dependency_relationship(
    target_entity_type: EntityType,
    source_entity_type: EntityType | None,
    *,
    target_column: str,
) -> None:
    """Validate one dependency edge against the entity relationship catalog.

    A missing source type is deferred until ``PiiReplacementPlan`` can infer it
    from the complete replacement graph; the target must still support
    dependencies at this earlier validation seam.
    """
    allowed = ALLOWED_DEPENDS_ON.get(target_entity_type, frozenset())
    if not allowed:
        raise ParameterError(
            f"column {target_column!r}: entity_type {target_entity_type.value!r} does not allow depends_on"
        )
    if source_entity_type is None or source_entity_type in allowed:
        return
    raise ParameterError(
        f"column {target_column!r}: depends_on entity_type "
        f"{source_entity_type.value!r} is not allowed for entity_type "
        f"{target_entity_type.value!r} (allowed: {sorted(entity.value for entity in allowed)})"
    )