Skip to content

patterns

patterns

Canonical descriptions of the supported PII replacement pattern grammars.

Classes:

Name Description
CharacterMaskToken

Regex, generated characters, and user-facing description for one character-mask token.

NamePartPlaceholder

Meaning and presentation of one accepted name-part placeholder.

CharacterMaskPart

One parsed literal or generated character in a character mask.

NamePatternPart

One parsed literal or placeholder in a name/email pattern.

Functions:

Name Description
parse_character_mask

Parse the character-mask grammar used by validation and generation.

compile_character_mask

Compile a character mask using the same parse tree used for rendering.

render_character_mask

Render one deterministic value from a valid character mask.

parse_name_pattern

Parse the shared whole-name and email pattern grammar.

compile_name_pattern

Compile a name/email pattern using the shared parser.

extract_name_components

Extract semantically named parts from a value matching a validated name pattern.

render_name_pattern

Render a valid name/email pattern from normalized component values.

pattern_grammar_catalog

Return structured grammar documentation keyed by pattern syntax name.

describe_pattern

Return each symbol in a valid pattern with its meaning, in order of first use.

CharacterMaskToken(regex, choices, description) dataclass

Regex, generated characters, and user-facing description for one character-mask token.

Attributes:

Name Type Description
regex str

Regex that validates one character produced by the token.

choices str

Characters the token can generate.

description str

Short description shown in prompts and error messages.

regex instance-attribute

Regex that validates one character produced by the token.

choices instance-attribute

Characters the token can generate.

description instance-attribute

Short description shown in prompts and error messages.

NamePartPlaceholder(part, capitalization, initial, description) dataclass

Meaning and presentation of one accepted name-part placeholder.

Attributes:

Name Type Description
part Literal['first', 'middle', 'last', 'domain', 'organization']

Name or email part the placeholder renders.

capitalization Literal['lower', 'title', 'upper'] | None

Case applied to the rendered part, or None to keep it as generated.

initial bool

Whether only the part's first character is rendered.

description str

Short description shown in prompts and error messages.

part instance-attribute

Name or email part the placeholder renders.

capitalization instance-attribute

Case applied to the rendered part, or None to keep it as generated.

initial instance-attribute

Whether only the part's first character is rendered.

description instance-attribute

Short description shown in prompts and error messages.

CharacterMaskPart(literal=None, choices=None) dataclass

One parsed literal or generated character in a character mask.

Attributes:

Name Type Description
literal str | None

Character copied unchanged, or None for a generated position.

choices str | None

Characters a generated position may produce, or None for a literal.

literal = None class-attribute instance-attribute

Character copied unchanged, or None for a generated position.

choices = None class-attribute instance-attribute

Characters a generated position may produce, or None for a literal.

NamePatternPart(literal=None, placeholder=None) dataclass

One parsed literal or placeholder in a name/email pattern.

Attributes:

Name Type Description
literal str | None

Text copied unchanged, or None for a placeholder.

placeholder NamePartPlaceholder | None

Placeholder rendered from generated name parts, or None for a literal.

literal = None class-attribute instance-attribute

Text copied unchanged, or None for a placeholder.

placeholder = None class-attribute instance-attribute

Placeholder rendered from generated name parts, or None for a literal.

parse_character_mask(pattern)

Parse the character-mask grammar used by validation and generation.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def parse_character_mask(pattern: str) -> tuple[tuple[CharacterMaskPart, ...] | None, str | None]:
    """Parse the character-mask grammar used by validation and generation."""
    parts: list[CharacterMaskPart] = []
    has_variable = False
    index = 0
    while index < len(pattern):
        character = pattern[index]
        if character == "\\":
            if index + 1 >= len(pattern):
                return None, "ends with a trailing '\\'"
            index += 1
            escaped = pattern[index]
            if escaped not in CHARACTER_MASK_ESCAPABLE_CHARACTERS:
                allowed = " ".join(sorted(CHARACTER_MASK_ESCAPABLE_CHARACTERS))
                return None, f"escapes unsupported character {escaped!r}; only {allowed} may be escaped"
            parts.append(CharacterMaskPart(literal=escaped))
        elif token := CHARACTER_MASK_TOKENS.get(character):
            parts.append(CharacterMaskPart(choices=token.choices))
            has_variable = True
        elif character == "[":
            choices, end, error = _parse_character_class(pattern, index)
            if error is not None:
                return None, error
            parts.append(CharacterMaskPart(choices=choices))
            has_variable = True
            index = end
        elif character == "]":
            return None, "has an unmatched ']'; escape it as '\\]' to write it literally"
        else:
            parts.append(CharacterMaskPart(literal=character))
        index += 1
    if not has_variable:
        return None, "has no variable placeholder"
    return tuple(parts), None

compile_character_mask(pattern)

Compile a character mask using the same parse tree used for rendering.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def compile_character_mask(pattern: str) -> tuple[re.Pattern[str] | None, str | None]:
    """Compile a character mask using the same parse tree used for rendering."""
    parts, error = parse_character_mask(pattern)
    if error is not None or parts is None:
        return None, error
    expression = "".join(
        re.escape(part.literal) if part.literal is not None else f"[{re.escape(part.choices or '')}]" for part in parts
    )
    return re.compile(expression), None

render_character_mask(pattern, rng)

Render one deterministic value from a valid character mask.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def render_character_mask(pattern: str, rng: Random) -> str:
    """Render one deterministic value from a valid character mask."""
    parts, error = parse_character_mask(pattern)
    if error is not None or parts is None:
        raise ParameterError(error or "invalid character mask")
    return "".join(part.literal if part.literal is not None else rng.choice(part.choices or "") for part in parts)

parse_name_pattern(entity_type, pattern)

Parse the shared whole-name and email pattern grammar.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def parse_name_pattern(
    entity_type: EntityType,
    pattern: str,
) -> tuple[tuple[NamePatternPart, ...] | None, str | None]:
    """Parse the shared whole-name and email pattern grammar."""
    matches = list(_NAME_PART_PATTERN.finditer(pattern))
    if not matches:
        return None, "has no name-part placeholder"
    unmatched = _NAME_PART_PATTERN.sub("", pattern)
    if "{" in unmatched or "}" in unmatched:
        return None, "has an unmatched '{' or '}'"

    parts: list[NamePatternPart] = []
    cursor = 0
    for match in matches:
        if literal := pattern[cursor : match.start()]:
            parts.append(NamePatternPart(literal=literal))
        placeholder = NAME_PART_PLACEHOLDERS.get(match.group(0))
        if placeholder is None:
            return None, f"uses unknown placeholder {match.group(0)!r}"
        if placeholder.part in _EMAIL_ONLY_NAME_PARTS and entity_type is not EntityType.EMAIL:
            return None, f"uses {match.group(0)} outside an email pattern"
        parts.append(NamePatternPart(placeholder=placeholder))
        cursor = match.end()
    if literal := pattern[cursor:]:
        parts.append(NamePatternPart(literal=literal))
    if entity_type is EntityType.EMAIL and "@" not in pattern:
        return None, "does not contain '@'"
    return tuple(parts), None

compile_name_pattern(entity_type, pattern)

Compile a name/email pattern using the shared parser.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def compile_name_pattern(
    entity_type: EntityType,
    pattern: str,
) -> tuple[re.Pattern[str] | None, str | None]:
    """Compile a name/email pattern using the shared parser."""
    parts, error = parse_name_pattern(entity_type, pattern)
    if error is not None or parts is None:
        return None, error
    expression: list[str] = []
    for part in parts:
        if part.literal is not None:
            expression.append(_name_literal_regex(part.literal, entity_type))
            continue
        placeholder = part.placeholder
        if placeholder is None:
            raise InternalError("parsed name pattern part has no literal or placeholder")
        expression.append(_name_placeholder_regex(placeholder, entity_type))
    return re.compile("".join(expression), re.UNICODE), None

extract_name_components(entity_type, pattern, value)

Extract semantically named parts from a value matching a validated name pattern.

The explicit placeholders provide the semantic alignment; this helper does not infer first, middle, or last names from arbitrary token positions. A pattern with two placeholders next to each other and no literal between them returns no components, because the boundary between them would be a guess: {First}{Last} could split AdaLovelace into AdaLovelac and e.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def extract_name_components(
    entity_type: EntityType,
    pattern: str,
    value: str,
) -> tuple[tuple[EntityType, str], ...]:
    """Extract semantically named parts from a value matching a validated name pattern.

    The explicit placeholders provide the semantic alignment; this helper does
    not infer first, middle, or last names from arbitrary token positions.
    A pattern with two placeholders next to each other and no literal between
    them returns no components, because the boundary between them would be a
    guess: ``{First}{Last}`` could split ``AdaLovelace`` into ``AdaLovelac``
    and ``e``.
    """
    if entity_type is not EntityType.FULL_NAME:
        return ()
    parts, error = parse_name_pattern(entity_type, pattern)
    if error is not None or parts is None or _has_adjacent_placeholders(parts):
        return ()

    expression: list[str] = []
    groups: list[tuple[str, EntityType]] = []
    for part in parts:
        if part.literal is not None:
            expression.append(re.escape(part.literal))
            continue
        placeholder = part.placeholder
        if placeholder is None or placeholder.part in _EMAIL_ONLY_NAME_PARTS:
            return ()
        group_name = f"part_{len(groups)}"
        expression.append(f"(?P<{group_name}>{_name_placeholder_regex(placeholder, entity_type)})")
        groups.append((group_name, _NAME_COMPONENT_TYPES[placeholder.part]))

    match = re.fullmatch("".join(expression), value, re.UNICODE)
    if match is None:
        return ()
    return tuple((component_type, match.group(group_name)) for group_name, component_type in groups)

render_name_pattern(entity_type, pattern, values, rng)

Render a valid name/email pattern from normalized component values.

In an email, each run of whitespace inside a name part becomes one dot, so the last name De La Cruz renders {first}.{last}@{domain} as kyle.de.la.cruz@example.com rather than an address containing spaces.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def render_name_pattern(
    entity_type: EntityType,
    pattern: str,
    values: Mapping[str, str],
    rng: Random,
) -> str:
    """Render a valid name/email pattern from normalized component values.

    In an email, each run of whitespace inside a name part becomes one dot, so the last name ``De La Cruz`` renders
    ``{first}.{last}@{domain}`` as ``kyle.de.la.cruz@example.com`` rather than an address containing spaces.
    """
    parts, error = parse_name_pattern(entity_type, pattern)
    if error is not None or parts is None:
        raise ParameterError(error or "invalid name pattern")
    rendered: list[str] = []
    for part in parts:
        if part.literal is not None:
            literal = part.literal
            if entity_type is EntityType.EMAIL:
                literal = "".join(str(rng.randrange(10)) if character == "#" else character for character in literal)
            rendered.append(literal)
            continue
        placeholder = part.placeholder
        if placeholder is None:
            raise InternalError("parsed name pattern part has no literal or placeholder")
        component = values.get(placeholder.part, "")
        if placeholder.initial:
            component = component[:1]
        if placeholder.capitalization == "upper":
            component = component.upper()
        elif placeholder.capitalization == "title":
            component = component.title()
        elif placeholder.capitalization == "lower":
            component = component.lower()
        if entity_type is EntityType.EMAIL and placeholder.part not in _EMAIL_ONLY_NAME_PARTS:
            component = ".".join(component.split())
        rendered.append(component)
    return "".join(rendered)

pattern_grammar_catalog()

Return structured grammar documentation keyed by pattern syntax name.

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def pattern_grammar_catalog() -> dict[str, dict[str, object]]:
    """Return structured grammar documentation keyed by pattern syntax name."""
    return {
        PatternSyntax.NAME_PARTS.name.lower(): {
            "description": "Whole-value templates composed of literals and name-part placeholders.",
            "placeholders": {
                placeholder: definition.description for placeholder, definition in NAME_PART_PLACEHOLDERS.items()
            },
            "rules": [
                "Literal separators and punctuation are preserved.",
                "{domain} may only be used for email.",
                "{organization} may only be used for email.",
                "Email patterns must contain @.",
                "In email patterns, # emits one digit.",
            ],
            "examples": ["{First} {Last}", "{f}.{last}@{domain}", "{first}@mail.{organization}.org"],
        },
        PatternSyntax.CHARACTER_MASK.name.lower(): {
            "description": "One generated character per variable token; all other characters are literal.",
            "tokens": {
                **{token: definition.description for token, definition in CHARACTER_MASK_TOKENS.items()},
                "[abc]": "one literal character from the brackets",
                "\\x": r"literal escaped special character; for example, \# matches #",
            },
            "rules": [
                "Bracket contents are literal choices, not ranges.",
                "Only " + ", ".join(_CHARACTER_MASK_ESCAPE_ORDER) + " may follow an escape character.",
                "The pattern must contain at least one variable token.",
            ],
            "examples": ["pmc-#######-#", "CUST-10[01]###", r"literal-\#-###"],
        },
        PatternSyntax.STRFTIME.name.lower(): {
            "description": "Python strftime/strptime format describing the complete datetime value.",
            "examples": ["%m/%d/%Y", "%Y-%m-%d"],
        },
    }

describe_pattern(entity_type, pattern)

Return each symbol in a valid pattern with its meaning, in order of first use.

Character masks list their tokens and bracket choices, and name patterns their placeholders, using the descriptions in CHARACTER_MASK_TOKENS and NAME_PART_PLACEHOLDERS. A strftime pattern uses Python's format codes, so it is described by an example date written in it.

Example

describe_pattern(EntityType.PHONE_NUMBER, "1-###") returns (("#", "digit 0-9"),), and describe_pattern(EntityType.DATE_OF_BIRTH, "%Y-%m-%d") returns (("%Y-%m-%d", "a date such as 1987-06-25"),).

Source code in src/nemo_safe_synthesizer/pii_replacer/planning/patterns.py
def describe_pattern(entity_type: EntityType, pattern: str) -> tuple[tuple[str, str], ...]:
    """Return each symbol in a valid ``pattern`` with its meaning, in order of first use.

    Character masks list their tokens and bracket choices, and name patterns their placeholders, using the
    descriptions in ``CHARACTER_MASK_TOKENS`` and ``NAME_PART_PLACEHOLDERS``. A strftime pattern uses Python's
    format codes, so it is described by an example date written in it.

    Example:
        ``describe_pattern(EntityType.PHONE_NUMBER, "1-###")`` returns ``(("#", "digit 0-9"),)``, and
        ``describe_pattern(EntityType.DATE_OF_BIRTH, "%Y-%m-%d")`` returns
        ``(("%Y-%m-%d", "a date such as 1987-06-25"),)``.
    """
    syntax = ENTITY_BY_TYPE[entity_type].pattern_syntax
    if syntax is PatternSyntax.STRFTIME:
        return ((pattern, f"a date such as {_STRFTIME_EXAMPLE.strftime(pattern)}"),)
    symbols: dict[str, str] = {}
    if syntax is PatternSyntax.NAME_PARTS:
        for match in _NAME_PART_PATTERN.finditer(pattern):
            if (placeholder := NAME_PART_PLACEHOLDERS.get(match.group(0))) is not None:
                symbols.setdefault(match.group(0), placeholder.description)
        if entity_type is EntityType.EMAIL and "#" in _NAME_PART_PATTERN.sub("", pattern):
            symbols.setdefault("#", CHARACTER_MASK_TOKENS["#"].description)
    elif syntax is PatternSyntax.CHARACTER_MASK:
        index = 0
        while index < len(pattern):
            character = pattern[index]
            if character == "\\":
                index += 2
                continue
            if character == "[" and (end := pattern.find("]", index + 1)) > index:
                symbols.setdefault(pattern[index : end + 1], f"one of {', '.join(pattern[index + 1 : end])}")
                index = end + 1
                continue
            if (token := CHARACTER_MASK_TOKENS.get(character)) is not None:
                symbols.setdefault(character, token.description)
            index += 1
    return tuple(symbols.items())