Skip to content

detection

detection

Free-text PII detection and deterministic span resolution.

Classes:

Name Description
FreeTextDetector

Detect and resolve PII spans for complete original dataframe cells.

Gliner2Detector

Lazy local GLiNER2 adapter with positional chunk remapping.

RegexDetector

Bounded deterministic detector for structurally validated PII.

CompositeFreeTextDetector

Combine GLiNER2 and regex detections through one overlap policy.

Functions:

Name Description
resolve_overlapping_spans

Resolve overlapping candidates, keeping longer spans first, then more confident ones.

CONTRACTION_SUFFIX = regex.compile("['\\u2019]t(?!\\w)", regex.IGNORECASE) module-attribute

The 't of a negative contraction such as don't, which never follows a name.

FreeTextDetector

Bases: Protocol

Detect and resolve PII spans for complete original dataframe cells.

Methods:

Name Description
detect

Return accepted, non-overlapping spans for cells and the spans replacement must leave unchanged.

detect(cells)

Return accepted, non-overlapping spans for cells and the spans replacement must leave unchanged.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def detect(self, cells: Sequence[DetectionCell]) -> FreeTextDetections:
    """Return accepted, non-overlapping spans for ``cells`` and the spans replacement must leave unchanged."""

Gliner2Detector(config, *, locale='en_US', model_loader=None)

Lazy local GLiNER2 adapter with positional chunk remapping.

The model is loaded on the first detect call and reused afterwards. Long texts are split into overlapping chunks, and detected offsets are mapped back to the original text.

Parameters:

Name Type Description Default
config FreeTextDetectionConfig

Detection settings: model ID and revision, entity and name-exclusion thresholds, chunk geometry, and batch size.

required
locale str

Locale whose country's personal IDs, checked by python-stdnum, use config.checked_national_id_threshold.

'en_US'
model_loader Callable[[str, str | None], _GlinerModel] | None

Callable that loads a model from a model ID and an optional revision. Defaults to loading the local GLiNER2 checkpoint; tests pass a fake.

None

Raises:

Type Description
ParameterError

If a name-exclusion threshold label is also a GLiNER2 detection label.

Methods:

Name Description
detect

Run GLiNER2 once per unique text and copy valid spans to matching cells.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def __init__(
    self,
    config: FreeTextDetectionConfig,
    *,
    locale: str = "en_US",
    model_loader: Callable[[str, str | None], _GlinerModel] | None = None,
) -> None:
    colliding = sorted(set(_exclusion_thresholds(config)) & set(_GLINER_LABELS))
    if colliding:
        raise ParameterError(
            f"free_text_detection.name_exclusion_thresholds cannot use GLiNER2 detection labels: {colliding}"
        )
    self._config = config
    self._is_national_id = national_id_check(locale)
    self._model_loader = model_loader or _load_gliner2_model
    self._model: _GlinerModel | None = None

detect(cells)

Run GLiNER2 once per unique text and copy valid spans to matching cells.

Within each unique text, name spans that overlap a name-exclusion span are dropped, then the middle-name neighbour rule is applied, both before per-cell entity filtering so every cell sees the same model decision. Name-exclusion spans are also returned as protected spans.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def detect(self, cells: Sequence[DetectionCell]) -> FreeTextDetections:
    """Run GLiNER2 once per unique text and copy valid spans to matching cells.

    Within each unique text, name spans that overlap a name-exclusion span
    are dropped, then the middle-name neighbour rule is applied, both before
    per-cell entity filtering so every cell sees the same model decision.
    Name-exclusion spans are also returned as protected spans.
    """
    cells_by_text = _cells_by_text(cells)
    if not cells_by_text:
        return FreeTextDetections(())

    chunks = list(chain.from_iterable(_chunks(text, self._config) for text in cells_by_text))
    results = self._infer(chunks, _requested_labels(cells, self._config, self._is_national_id))
    candidates: dict[str, _TextCandidates] = defaultdict(_TextCandidates)
    rejected_birth_dates = invalid_spans = 0
    for chunk, result in zip(chunks, results, strict=True):
        rejected = _collect_chunk_candidates(
            chunk, result, candidates[chunk.original_text], self._config, self._is_national_id
        )
        rejected_birth_dates += rejected.birth_dates
        invalid_spans += rejected.invalid_spans
    spans: list[DetectedSpan] = []
    protected: list[ProtectedSpan] = []
    for text, text_cells in cells_by_text.items():
        for span in _filter_name_spans(candidates[text], text):
            spans.extend(_copy_span_to_cells(span, text_cells))
        # Overlapping chunks can report the same exclusion span twice.
        for start, end in sorted(set(candidates[text].exclusions)):
            protected.extend(ProtectedSpan(cell.cell_id, start, end) for cell in text_cells)
    if rejected_birth_dates:
        logger.user.warning(
            "GLiNER2 birth-date candidates were ignored because they were not complete parseable dates",
            extra={"rejected_candidate_count": rejected_birth_dates},
        )
    if invalid_spans:
        logger.user.warning(
            "GLiNER2 spans were ignored because their offsets, score, or metadata were invalid",
            extra={"invalid_span_count": invalid_spans},
        )
    return FreeTextDetections(tuple(spans), tuple(protected))

RegexDetector(*, timeout_seconds=_DEFAULT_REGEX_TIMEOUT_SECONDS, max_matches_per_rule=_DEFAULT_MAX_MATCHES_PER_RULE)

Bounded deterministic detector for structurally validated PII.

Each built-in rule runs with a timeout and a cap on matches per cell, so an adversarial cell cannot stall or flood detection, and each match must pass the rule's structural validator, such as a Luhn check for card numbers, to become a span.

Parameters:

Name Type Description Default
timeout_seconds float

Time limit for one rule on one cell.

_DEFAULT_REGEX_TIMEOUT_SECONDS
max_matches_per_rule int

Largest number of matches one rule may produce for one cell before detection fails.

_DEFAULT_MAX_MATCHES_PER_RULE

Raises:

Type Description
InternalError

If timeout_seconds or max_matches_per_rule is not positive.

Methods:

Name Description
detect

Return exact spans from applicable built-in rules. Regex rules protect nothing.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def __init__(
    self,
    *,
    timeout_seconds: float = _DEFAULT_REGEX_TIMEOUT_SECONDS,
    max_matches_per_rule: int = _DEFAULT_MAX_MATCHES_PER_RULE,
) -> None:
    if timeout_seconds <= 0:
        raise InternalError("regex detector timeout_seconds must be positive")
    if max_matches_per_rule <= 0:
        raise InternalError("regex detector max_matches_per_rule must be positive")
    self._timeout_seconds = timeout_seconds
    self._max_matches_per_rule = max_matches_per_rule

detect(cells)

Return exact spans from applicable built-in rules. Regex rules protect nothing.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def detect(self, cells: Sequence[DetectionCell]) -> FreeTextDetections:
    """Return exact spans from applicable built-in rules. Regex rules protect nothing."""
    return FreeTextDetections(tuple(chain.from_iterable(self._detect_cell(cell) for cell in cells)))

CompositeFreeTextDetector(config, *, locale='en_US', gliner=None, regex_detector=None)

Combine GLiNER2 and regex detections through one overlap policy.

Methods:

Name Description
detect

Return globally resolved spans and the protected spans from both mandatory detector sources.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def __init__(
    self,
    config: FreeTextDetectionConfig,
    *,
    locale: str = "en_US",
    gliner: FreeTextDetector | None = None,
    regex_detector: FreeTextDetector | None = None,
) -> None:
    self._gliner = gliner or Gliner2Detector(config, locale=locale)
    self._regex = regex_detector or RegexDetector()

detect(cells)

Return globally resolved spans and the protected spans from both mandatory detector sources.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def detect(self, cells: Sequence[DetectionCell]) -> FreeTextDetections:
    """Return globally resolved spans and the protected spans from both mandatory detector sources."""
    gliner = self._gliner.detect(cells)
    regex = self._regex.detect(cells)
    by_cell: dict[DetectionCellId, list[DetectedSpan]] = defaultdict(list)
    for span in (*gliner.spans, *regex.spans):
        by_cell[span.cell_id].append(span)
    accepted: list[DetectedSpan] = []
    for cell in cells:
        accepted.extend(resolve_overlapping_spans(by_cell.get(cell.cell_id, ())))
    return FreeTextDetections(tuple(accepted), (*gliner.protected, *regex.protected))

resolve_overlapping_spans(spans)

Resolve overlapping candidates, keeping longer spans first, then more confident ones.

Spans are ranked by length, then confidence (regex spans count as 1.0 and known-value matches as 0.0), then position, then source, GLiNER2 score, and entity order, and each is kept unless it overlaps an already kept span.

Example

Two overlapping five-character spans at offsets 0 and 2 with scores 0.6 and 0.9 keep the span at offset 2.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def resolve_overlapping_spans(spans: Iterable[DetectedSpan]) -> tuple[DetectedSpan, ...]:
    """Resolve overlapping candidates, keeping longer spans first, then more confident ones.

    Spans are ranked by length, then confidence (regex spans count as 1.0 and
    known-value matches as 0.0), then position, then source, GLiNER2 score, and entity order, and each is
    kept unless it overlaps an already kept span.

    Example:
        Two overlapping five-character spans at offsets 0 and 2 with scores
        0.6 and 0.9 keep the span at offset 2.
    """
    best_duplicates: dict[tuple[DetectionCellId, EntityType, int, int], DetectedSpan] = {}
    for span in spans:
        key = (span.cell_id, span.entity_type, span.start, span.end)
        current = best_duplicates.get(key)
        if current is None or _tie_rank(span) < _tie_rank(current):
            best_duplicates[key] = span

    ranked = sorted(
        best_duplicates.values(),
        key=lambda span: (
            -(span.end - span.start),
            -_confidence(span),
            span.start,
            span.end,
            *_tie_rank(span),
        ),
    )
    accepted: list[DetectedSpan] = []
    for candidate in ranked:
        if not any(_overlaps(candidate, existing) for existing in accepted):
            accepted.append(candidate)
    return tuple(
        sorted(accepted, key=lambda span: (span.cell_id.row_position, span.start, span.end, span.entity_type.value))
    )

fresh_detection_entity_types()

Return entity types eligible for fresh replacement inside free text.

Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
def fresh_detection_entity_types() -> frozenset[EntityType]:
    """Return entity types eligible for fresh replacement inside free text."""
    return _FRESH_ENTITY_TYPES