Skip to content

free_text

free_text

Detection, scoped mapping reuse, and one-pass free-text replacement.

Classes:

Name Description
FreeTextColumnStatistics

Mutable aggregate statistics for one planned free-text column.

FreeTextCellDetections

Validated detector output for the original free-text cells, grouped by cell.

FreeTextExecutionResult

Aggregate outputs owned by one free-text execution pass.

FreeTextReplacementExecutor

Replace accepted spans while preserving scoped structured mappings.

FreeTextColumnStatistics(counts=dict(), values=dict(), sources=set(), kept_name_part_count=0) dataclass

Mutable aggregate statistics for one planned free-text column.

Attributes:

Name Type Description
counts dict[str, int]

Accepted span count per entity type.

values dict[str, set[str]]

Distinct accepted span texts per entity type.

sources set[str]

Detector sources that produced accepted spans.

kept_name_part_count int

Full-name replacements that kept an original first or last name because no other name was found.

counts = field(default_factory=dict) class-attribute instance-attribute

Accepted span count per entity type.

values = field(default_factory=dict) class-attribute instance-attribute

Distinct accepted span texts per entity type.

sources = field(default_factory=set) class-attribute instance-attribute

Detector sources that produced accepted spans.

kept_name_part_count = 0 class-attribute instance-attribute

Full-name replacements that kept an original first or last name because no other name was found.

FreeTextCellDetections(spans=dict(), protected=dict()) dataclass

Validated detector output for the original free-text cells, grouped by cell.

Attributes:

Name Type Description
spans dict[DetectionCellId, tuple[DetectedSpan, ...]]

Accepted, non-overlapping spans to replace.

protected dict[DetectionCellId, tuple[ProtectedSpan, ...]]

Spans that known-value matches must not overlap.

spans = field(default_factory=dict) class-attribute instance-attribute

Accepted, non-overlapping spans to replace.

protected = field(default_factory=dict) class-attribute instance-attribute

Spans that known-value matches must not overlap.

FreeTextExecutionResult(column_statistics, generated_replacement_count, generator_time_seconds, replacement_records) dataclass

Aggregate outputs owned by one free-text execution pass.

Attributes:

Name Type Description
column_statistics dict[str, FreeTextColumnStatistics]

Detection statistics keyed by free-text column name.

generated_replacement_count int

Replacements generated fresh rather than reused from a mapping.

generator_time_seconds float

Time spent inside the generator for free-text spans.

replacement_records tuple[FreeTextReplacementRecord, ...]

Per-span records when capture was requested, otherwise empty.

column_statistics instance-attribute

Detection statistics keyed by free-text column name.

generated_replacement_count instance-attribute

Replacements generated fresh rather than reused from a mapping.

generator_time_seconds instance-attribute

Time spent inside the generator for free-text spans.

replacement_records instance-attribute

Per-span records when capture was requested, otherwise empty.

FreeTextReplacementExecutor(generator, detector, *, base_seed, capture_replacement_map=False, find_and_replace=False, locale='en_US')

Replace accepted spans while preserving scoped structured mappings.

Detection always reads complete original cells before structured columns mutate. Replacement later uses those original offsets to build each cell exactly once. Structured mappings are registered through the same module so independently detected exact values and validated name components remain consistent within record or group scope. With find_and_replace, every cell is also searched for those structured and detected values, so occurrences the detector missed get the same replacement.

Parameters:

Name Type Description Default
generator ReplacementGenerator

Produces a replacement for each value that has no mapping yet.

required
detector FreeTextDetector | None

Finds candidate spans in free-text cells, or None when the plan has no free-text targets; detect then fails if called with any.

required
base_seed int

Seed from which every replacement's deterministic seed is derived.

required
capture_replacement_map bool

Whether to keep a per-span record of each original and replacement value in the execution result.

False
find_and_replace bool

Whether to also replace occurrences of known structured and detected values that the detector missed.

False
locale str

replace_pii.locale, whose name order full names in text are read in.

'en_US'

Methods:

Name Description
reset

Discard all mappings and aggregates from an earlier dataframe execution.

detect

Detect and validate candidate and protected spans against complete original cells.

execute

Apply accepted original-cell spans to every planned free-text target.

detect_statistics

Record the spans execute would replace in each free-text target, without replacing them.

register_structured_mapping

Expose exact values, pattern-validated name parts, and street parts to accepted spans.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/free_text.py
def __init__(
    self,
    generator: ReplacementGenerator,
    detector: FreeTextDetector | None,
    *,
    base_seed: int,
    capture_replacement_map: bool = False,
    find_and_replace: bool = False,
    locale: str = "en_US",
) -> None:
    self._generator = generator
    self._detector = detector
    self._locale = locale
    self._base_seed = base_seed
    self._capture_replacement_map = capture_replacement_map
    self._find_and_replace = find_and_replace
    self._cache: dict[FreeTextMappingKey, str] = {}
    # Cached name-part replacements by scope, part type, and the original word's case-folded first letter,
    # so an initial such as "L" can reuse the replacement of the one "Lovelace" seen in its scope.
    self._name_words_by_initial: dict[tuple[Hashable, EntityType, str], dict[str, str]] = {}
    self._statistics: dict[str, FreeTextColumnStatistics] = {}
    self._replacement_records: list[FreeTextReplacementRecord] = []
    self._generation_elapsed = 0.0
    self._generated_count = 0

reset()

Discard all mappings and aggregates from an earlier dataframe execution.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/free_text.py
def reset(self) -> None:
    """Discard all mappings and aggregates from an earlier dataframe execution."""
    self._cache.clear()
    self._name_words_by_initial.clear()
    self._statistics.clear()
    self._replacement_records.clear()
    self._generation_elapsed = 0.0
    self._generated_count = 0

detect(dataframe, specs)

Detect and validate candidate and protected spans against complete original cells.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/free_text.py
def detect(
    self,
    dataframe: pd.DataFrame,
    specs: list[PiiColumnPlan],
) -> FreeTextCellDetections:
    """Detect and validate candidate and protected spans against complete original cells."""
    if not specs:
        return FreeTextCellDetections()
    if self._detector is None:
        raise GenerationError(
            "free-text replacement targets require the detector and span-resolution implementation"
        )

    cells = list(chain.from_iterable(_detection_cells(dataframe, spec) for spec in specs))
    requested = {cell.cell_id: cell for cell in cells}
    try:
        detected = self._detector.detect(cells)
        candidates = _validated_candidates(detected.spans, requested)
        protected = _validated_protected(detected.protected, requested)
    except SafeSynthesizerError:
        raise
    except Exception as exc:
        raise GenerationError("free-text PII detection returned invalid spans") from exc
    return FreeTextCellDetections(
        {cell_id: resolve_overlapping_spans(spans) for cell_id, spans in candidates.items()},
        {cell_id: tuple(spans) for cell_id, spans in protected.items()},
    )

execute(working, specs, group_identities, detections)

Apply accepted original-cell spans to every planned free-text target.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/free_text.py
def execute(
    self,
    working: pd.DataFrame,
    specs: list[PiiColumnPlan],
    group_identities: tuple[CanonicalValue, ...] | None,
    detections: FreeTextCellDetections,
) -> FreeTextExecutionResult:
    """Apply accepted original-cell spans to every planned free-text target."""
    known_values = (
        self._known_values(working, specs, group_identities, detections.spans) if self._find_and_replace else {}
    )
    cells = {
        spec.column_name: self._cells_to_replace(working, spec, group_identities, detections, known_values)
        for spec in specs
    }
    # Every whole name word in a record or group is mapped before any initial is written, so an
    # initial takes its name's replacement letter wherever the name appears.
    for spec in specs:
        self._map_name_words(cells[spec.column_name], self._statistics_for(spec))
    scope = MappingScope.RECORD if group_identities is None else MappingScope.GROUP
    for spec in specs:
        self._write_cells(working, spec, cells[spec.column_name], scope)
    return FreeTextExecutionResult(
        column_statistics=self._statistics,
        generated_replacement_count=self._generated_count,
        generator_time_seconds=self._generation_elapsed,
        replacement_records=tuple(self._replacement_records),
    )

detect_statistics(dataframe, specs, group_identities, detections)

Record the spans execute would replace in each free-text target, without replacing them.

Structured mappings must be registered first, as execute expects, so known-value matches are the same.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/free_text.py
def detect_statistics(
    self,
    dataframe: pd.DataFrame,
    specs: list[PiiColumnPlan],
    group_identities: tuple[CanonicalValue, ...] | None,
    detections: FreeTextCellDetections,
) -> dict[str, FreeTextColumnStatistics]:
    """Record the spans ``execute`` would replace in each free-text target, without replacing them.

    Structured mappings must be registered first, as ``execute`` expects, so known-value matches are the same.
    """
    known_values = (
        self._known_values(dataframe, specs, group_identities, detections.spans) if self._find_and_replace else {}
    )
    for spec in specs:
        statistics = self._statistics_for(spec)
        for to_replace in self._cells_to_replace(dataframe, spec, group_identities, detections, known_values):
            _record_cell_detections(statistics, to_replace)
    return self._statistics

register_structured_mapping(key, spec, original_text, replacement, address_parts=None)

Expose exact values, pattern-validated name parts, and street parts to accepted spans.

original_text is the source cell's text form, the same value the structured generator received, so a typed original such as an integer API key matches the digits a detector finds in prose. key carries its typed identity. address_parts is the detected street of a street-address value whose locality was kept.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/free_text.py
def register_structured_mapping(
    self,
    key: RecordMappingKey | GroupMappingKey,
    spec: PiiColumnPlan,
    original_text: str,
    replacement: str,
    address_parts: AddressParts | None = None,
) -> None:
    """Expose exact values, pattern-validated name parts, and street parts to accepted spans.

    ``original_text`` is the source cell's text form, the same value the
    structured generator received, so a typed original such as an integer
    API key matches the digits a detector finds in prose. ``key`` carries
    its typed identity. ``address_parts`` is the detected street of a
    street-address value whose locality was kept.
    """
    scope_identity = key.row_position if isinstance(key, RecordMappingKey) else key.original_group_identity
    self._register(scope_identity, spec.entity_type, original_text, replacement)
    self._register_name_components(scope_identity, spec, original_text, replacement)
    self._register_address_component(scope_identity, spec, replacement, address_parts)