detection
detection
¶
Free-text PII detection and deterministic span resolution.
Classes:
| Name | Description |
|---|---|
FreeTextDetector |
Detect and resolve PII spans for complete original dataframe cells. |
Gliner2Detector |
Lazy local GLiNER2 adapter with positional chunk remapping. |
RegexDetector |
Bounded deterministic detector for structurally validated PII. |
CompositeFreeTextDetector |
Combine GLiNER2 and regex detections through one overlap policy. |
Functions:
| Name | Description |
|---|---|
resolve_overlapping_spans |
Resolve overlapping candidates, keeping longer spans first, then more confident ones. |
CONTRACTION_SUFFIX = regex.compile("['\\u2019]t(?!\\w)", regex.IGNORECASE)
module-attribute
¶
The 't of a negative contraction such as don't, which never follows a name.
FreeTextDetector
¶
Bases: Protocol
Detect and resolve PII spans for complete original dataframe cells.
Methods:
| Name | Description |
|---|---|
detect |
Return accepted, non-overlapping spans for |
detect(cells)
¶
Return accepted, non-overlapping spans for cells and the spans replacement must leave unchanged.
Gliner2Detector(config, *, locale='en_US', model_loader=None)
¶
Lazy local GLiNER2 adapter with positional chunk remapping.
The model is loaded on the first detect call and reused afterwards.
Long texts are split into overlapping chunks, and detected offsets are
mapped back to the original text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
FreeTextDetectionConfig
|
Detection settings: model ID and revision, entity and name-exclusion thresholds, chunk geometry, and batch size. |
required |
locale
|
str
|
Locale whose country's personal IDs, checked by python-stdnum,
use |
'en_US'
|
model_loader
|
Callable[[str, str | None], _GlinerModel] | None
|
Callable that loads a model from a model ID and an optional revision. Defaults to loading the local GLiNER2 checkpoint; tests pass a fake. |
None
|
Raises:
| Type | Description |
|---|---|
ParameterError
|
If a name-exclusion threshold label is also a GLiNER2 detection label. |
Methods:
| Name | Description |
|---|---|
detect |
Run GLiNER2 once per unique text and copy valid spans to matching cells. |
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
detect(cells)
¶
Run GLiNER2 once per unique text and copy valid spans to matching cells.
Within each unique text, name spans that overlap a name-exclusion span are dropped, then the middle-name neighbour rule is applied, both before per-cell entity filtering so every cell sees the same model decision. Name-exclusion spans are also returned as protected spans.
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
RegexDetector(*, timeout_seconds=_DEFAULT_REGEX_TIMEOUT_SECONDS, max_matches_per_rule=_DEFAULT_MAX_MATCHES_PER_RULE)
¶
Bounded deterministic detector for structurally validated PII.
Each built-in rule runs with a timeout and a cap on matches per cell, so an adversarial cell cannot stall or flood detection, and each match must pass the rule's structural validator, such as a Luhn check for card numbers, to become a span.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
timeout_seconds
|
float
|
Time limit for one rule on one cell. |
_DEFAULT_REGEX_TIMEOUT_SECONDS
|
max_matches_per_rule
|
int
|
Largest number of matches one rule may produce for one cell before detection fails. |
_DEFAULT_MAX_MATCHES_PER_RULE
|
Raises:
| Type | Description |
|---|---|
InternalError
|
If |
Methods:
| Name | Description |
|---|---|
detect |
Return exact spans from applicable built-in rules. Regex rules protect nothing. |
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
detect(cells)
¶
Return exact spans from applicable built-in rules. Regex rules protect nothing.
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
CompositeFreeTextDetector(config, *, locale='en_US', gliner=None, regex_detector=None)
¶
Combine GLiNER2 and regex detections through one overlap policy.
Methods:
| Name | Description |
|---|---|
detect |
Return globally resolved spans and the protected spans from both mandatory detector sources. |
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
detect(cells)
¶
Return globally resolved spans and the protected spans from both mandatory detector sources.
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
resolve_overlapping_spans(spans)
¶
Resolve overlapping candidates, keeping longer spans first, then more confident ones.
Spans are ranked by length, then confidence (regex spans count as 1.0 and known-value matches as 0.0), then position, then source, GLiNER2 score, and entity order, and each is kept unless it overlaps an already kept span.
Example
Two overlapping five-character spans at offsets 0 and 2 with scores 0.6 and 0.9 keep the span at offset 2.
Source code in src/nemo_safe_synthesizer/pii_replacer/replacement/detection.py
fresh_detection_entity_types()
¶
Return entity types eligible for fresh replacement inside free text.