Skip to content

window_group_mia

window_group_mia

Window membership inference protection for grouped numeric time-series data.

The metric asks whether synthetic data sits closer to training sequences than to unseen holdout sequences from the same process, the trace a generator leaves when it memorizes training data. Short windows of consecutive values, across all evaluated numeric columns at once, are compared with their nearest synthetic window. The score measures how much more often training windows are flagged as close than holdout windows are, after removing the difference expected from chance, on a 0--10 scale where 10 means no measurable memorization and 0 means every training window is flagged.

Classes:

Name Description
WindowGroupMIA

Measure how well synthetic data protects training sequences from window membership inference.

WindowGroupMIA pydantic-model

Bases: TimeSeriesComponent

Measure how well synthetic data protects training sequences from window membership inference.

Training and holdout windows are compared with their nearest synthetic window. A window is flagged when it is closer than all but the closest 5% of holdout windows. The privacy loss is the log ratio of the training and holdout flag rates. Chance variation from the binomial counts and from the holdout groups' own offset is removed before scoring 10 * (exp(-privacy_loss) - alpha) / (1 - alpha), where alpha is the holdout flag rate, so 10 means equal flag rates and 0 means every training window is flagged. Exact copies beyond the holdout's own exact-match rate set a floor on the privacy loss that chance can't remove, so a small holdout can't hide them. Without a group column or a usable holdout of at least two groups, the score is unavailable and only copy diagnostics are reported. With one holdout group, chance alone can cancel the privacy loss of blurred copies and report a perfect score.

Attributes:

Name Type Description
name str

Display name used in serialized evaluation results.

details dict[str, Any]

Flag rates, copy diagnostics, window counts, distances, and per-group summaries.

Fields:

name = 'Window Membership Inference Protection' pydantic-field

Display name used in serialized evaluation results.

details pydantic-field

Flag rates, copy diagnostics, window counts, distances, and per-group summaries.

jinja_context cached property

Return score, details, card attributes, and chart figures for report rendering.

from_evaluation_datasets(evaluation_datasets, config=None) staticmethod

Evaluate window membership inference protection for paired time-series datasets.

Report orchestration controls whether the optional metric runs. Calling this component directly always computes it with the supplied configuration or isolated defaults. Evaluation failures are returned in score.notes instead of aborting the full evaluation pipeline.

Parameters:

Name Type Description Default
evaluation_datasets EvaluationDatasets

Training, synthetic, and holdout (test) datasets.

required
config SafeSynthesizerParameters | None

Optional pipeline and metric configuration.

None

Returns:

Type Description
WindowGroupMIA

A component containing the score, diagnostic details, and notes.

Source code in src/nemo_safe_synthesizer/evaluation/components/window_group_mia.py
@staticmethod
def from_evaluation_datasets(
    evaluation_datasets: EvaluationDatasets,
    config: SafeSynthesizerParameters | None = None,
) -> WindowGroupMIA:
    """Evaluate window membership inference protection for paired time-series datasets.

    Report orchestration controls whether the optional metric runs.
    Calling this component directly always computes it with the supplied
    configuration or isolated defaults. Evaluation failures are returned
    in ``score.notes`` instead of aborting the full evaluation pipeline.

    Args:
        evaluation_datasets: Training, synthetic, and holdout (``test``) datasets.
        config: Optional pipeline and metric configuration.

    Returns:
        A component containing the score, diagnostic details, and notes.
    """
    cfg = WindowGroupMIA._resolve_config(config)
    # Optional metrics must fail independently so one diagnostic cannot
    # prevent the rest of the evaluation report from being produced.
    try:
        return WindowGroupMIA._evaluate(evaluation_datasets, cfg, config)
    except Exception as exc:
        logger.exception("Failed to compute Window Membership Inference Protection.")
        return WindowGroupMIA(score=EvaluationScore(notes=str(exc)))