Skip to content

window_group_mia_figures

window_group_mia_figures

Plotly report figures for window membership inference protection.

Each builder summarizes values stored by the metric in WindowGroupMIA.details.

Functions:

Name Description
generate_window_group_mia_distance_figure

Plot the distribution of each window's distance to its nearest synthetic window.

generate_window_group_mia_group_figure

Plot the share of each training group's windows that are flagged, for the 12 most-flagged groups.

generate_window_group_mia_closest_match_figure

Plot the synthetic window closest to the training data, with its training window for exact copies.

generate_window_group_mia_distance_figure(details)

Plot the distribution of each window's distance to its nearest synthetic window.

Distances are shown as the typical difference per value, in training standard deviations: the Euclidean distance divided by the square root of the number of values in a window. Training windows are drawn in training blue, holdout windows in gray, and the flag threshold as a dashed line. When training windows pile up to the left of the holdout windows, the synthetic data sits closer to training data than to unseen data. The holdout trace is not named after a dataset, so the report's dataset theming leaves its color alone.

Parameters:

Name Type Description Default
details Mapping[str, Any]

WindowGroupMIA.details with distances and, for a full evaluation, attack.

required

Returns:

Type Description
Figure

A Plotly overlaid histogram figure.

Raises:

Type Description
DataError

If no training distances are stored.

Source code in src/nemo_safe_synthesizer/evaluation/components/window_group_mia_figures.py
def generate_window_group_mia_distance_figure(details: Mapping[str, Any]) -> go.Figure:
    """Plot the distribution of each window's distance to its nearest synthetic window.

    Distances are shown as the typical difference per value, in training
    standard deviations: the Euclidean distance divided by the square root of
    the number of values in a window. Training windows are drawn in training
    blue, holdout windows in gray, and the flag threshold as a dashed line.
    When training windows pile up to the left of the holdout windows, the
    synthetic data sits closer to training data than to unseen data. The holdout
    trace is not named after a dataset, so the report's dataset theming leaves
    its color alone.

    Args:
        details: ``WindowGroupMIA.details`` with ``distances`` and, for a full
            evaluation, ``attack``.

    Returns:
        A Plotly overlaid histogram figure.

    Raises:
        DataError: If no training distances are stored.
    """
    distances = details.get("distances", {})
    training = np.asarray(distances.get("training", []), dtype=float)
    if training.size == 0:
        raise DataError("At least one training window distance is required.")
    holdout = np.asarray(distances.get("holdout", []), dtype=float)
    scale = float(np.sqrt(details["window_size"] * len(details["columns"])))
    attack = details.get("attack")
    # Clip each distribution's tail separately, so many exact training copies
    # can't push the holdout windows or the threshold off the chart.
    limits = [
        float(np.quantile(values / scale, _DISTANCE_RANGE_QUANTILE)) for values in (training, holdout) if values.size
    ]
    if attack is not None:
        limits.append(attack["threshold"] / scale)
    top = max([*limits, 1e-6]) * 1.05
    bins = {"start": 0.0, "end": top, "size": top / _DISTANCE_BINS}
    figure = go.Figure()
    figure.add_trace(
        go.Histogram(
            x=training / scale,
            xbins=bins,
            histnorm="percent",
            name="Training windows",
            marker={"color": _TRAINING_COLOR},
            opacity=0.65,
            hovertemplate="Training windows<br>difference %{x}<br>%{y:.1f}% of windows<extra></extra>",
        )
    )
    if holdout.size:
        figure.add_trace(
            go.Histogram(
                x=holdout / scale,
                xbins=bins,
                histnorm="percent",
                name="Holdout windows",
                marker={"color": _HOLDOUT_COLOR},
                opacity=0.65,
                hovertemplate="Holdout windows<br>difference %{x}<br>%{y:.1f}% of windows<extra></extra>",
            )
        )
    if attack is not None:
        figure.add_vline(
            x=attack["threshold"] / scale,
            line={"color": "rgba(255,255,255,0.6)", "dash": "dash", "width": 1},
            annotation_text="flag threshold",
            annotation_position="top",
        )
    figure.update_layout(
        template="plotly_white",
        height=_FIGURE_HEIGHT,
        barmode="overlay",
        showlegend=False,
        xaxis_title="Difference per value (training sd)",
        yaxis_title="Share of windows (%)",
        margin={"l": 54, "r": 16, "t": 40, "b": 48},
    )
    figure.update_xaxes(range=[0.0, top])
    return figure

generate_window_group_mia_group_figure(per_group, holdout_flag_rate)

Plot the share of each training group's windows that are flagged, for the 12 most-flagged groups.

Bars shade from white at the holdout flag rate, which is what chance alone produces, to red when every window of the group is flagged. The holdout flag rate is drawn as a dashed line, and the x-axis always spans 0 to 100% so small excesses look small. Group names are shortened only in the axis labels; hover shows the full name, window count, and exact copies.

Parameters:

Name Type Description Default
per_group Sequence[Mapping[str, Any]]

WindowGroupMIA.details["per_group"] from a full evaluation.

required
holdout_flag_rate float

Share of holdout windows flagged.

required

Returns:

Type Description
Figure

A Plotly horizontal bar figure.

Raises:

Type Description
DataError

If no group summaries with flag rates are supplied.

Source code in src/nemo_safe_synthesizer/evaluation/components/window_group_mia_figures.py
def generate_window_group_mia_group_figure(
    per_group: Sequence[Mapping[str, Any]], holdout_flag_rate: float
) -> go.Figure:
    """Plot the share of each training group's windows that are flagged, for the 12 most-flagged groups.

    Bars shade from white at the holdout flag rate, which is what chance
    alone produces, to red when every window of the group is flagged. The
    holdout flag rate is drawn as a dashed line, and the x-axis always spans 0
    to 100% so small excesses look small. Group names are shortened only in the
    axis labels; hover shows the full name, window count, and exact copies.

    Args:
        per_group: ``WindowGroupMIA.details["per_group"]`` from a full evaluation.
        holdout_flag_rate: Share of holdout windows flagged.

    Returns:
        A Plotly horizontal bar figure.

    Raises:
        DataError: If no group summaries with flag rates are supplied.
    """
    flagged = [item for item in per_group if "flag_rate" in item]
    if not flagged:
        raise DataError("At least one group summary with a flag rate is required.")
    rows = sorted(flagged, key=lambda item: (-float(item["flag_rate"]), str(item["group"])))[:_GROUP_MAX_ROWS]
    rows.reverse()
    rates = np.array([float(item["flag_rate"]) for item in rows])
    excess = np.clip((rates - holdout_flag_rate) / max(1.0 - holdout_flag_rate, 1e-12), 0.0, 1.0)
    labels = ["all rows" if item["group"] is None else str(item["group"]) for item in rows]
    figure = go.Figure(
        go.Bar(
            x=100 * rates,
            y=list(range(len(rows))),
            orientation="h",
            customdata=[
                [label, item["windows"], 100 * float(item["exact_copy_rate"])]
                for label, item in zip(labels, rows, strict=True)
            ],
            marker={
                "color": excess,
                "colorscale": _DIFFERENCE_COLORSCALE,
                "cmin": 0.0,
                "cmax": 1.0,
                "line": {"color": "rgba(255,255,255,0.5)", "width": 1},
            },
            name="Flagged share",
            hovertemplate=(
                "group %{customdata[0]}<br>%{x:.1f}% of %{customdata[1]} windows flagged"
                "<br>%{customdata[2]:.1f}% exact copies<extra></extra>"
            ),
        )
    )
    figure.add_vline(
        x=100 * holdout_flag_rate,
        line={"color": "rgba(255,255,255,0.6)", "dash": "dash", "width": 1},
        annotation_text="holdout",
        annotation_position="top",
    )
    figure.update_layout(
        template="plotly_white",
        height=_FIGURE_HEIGHT,
        showlegend=False,
        xaxis_title="Windows flagged (%)",
        margin={"l": 80, "r": 16, "t": 40, "b": 48},
    )
    figure.update_xaxes(range=[0, 100])
    figure.update_yaxes(
        range=[-0.6, len(rows) - 0.4],
        tickmode="array",
        tickvals=list(range(len(rows))),
        ticktext=[shorten_column_label(label) for label in labels],
        zeroline=False,
    )
    return figure

generate_window_group_mia_closest_match_figure(details)

Plot the synthetic window closest to the training data, with its training window for exact copies.

The training window is plotted only when the match stores its values, which WindowGroupMIA does only for exact copies, so the report never shows training values that the synthetic data doesn't already contain. Each value column contributes two consecutive traces, training then synthetic, or only the synthetic trace when the training window is hidden. Columns follow details["columns"] order, and only the first column is visible so the report can switch columns by trace index.

Parameters:

Name Type Description Default
details Mapping[str, Any]

WindowGroupMIA.details with closest_match and columns.

required

Returns:

Type Description
Figure

A Plotly line figure with one visible column at a time.

Raises:

Type Description
DataError

If no closest match is stored.

Source code in src/nemo_safe_synthesizer/evaluation/components/window_group_mia_figures.py
def generate_window_group_mia_closest_match_figure(details: Mapping[str, Any]) -> go.Figure:
    """Plot the synthetic window closest to the training data, with its training window for exact copies.

    The training window is plotted only when the match stores its values,
    which ``WindowGroupMIA`` does only for exact copies, so the report never
    shows training values that the synthetic data doesn't already contain.
    Each value column contributes two consecutive traces, training then
    synthetic, or only the synthetic trace when the training window is
    hidden. Columns follow ``details["columns"]`` order, and only the first
    column is visible so the report can switch columns by trace index.

    Args:
        details: ``WindowGroupMIA.details`` with ``closest_match`` and ``columns``.

    Returns:
        A Plotly line figure with one visible column at a time.

    Raises:
        DataError: If no closest match is stored.
    """
    match = details.get("closest_match")
    if not match:
        raise DataError("A closest training and synthetic window match is required.")
    steps = np.arange(1, details["window_size"] + 1)
    figure = go.Figure()
    windows = [("synthetic_values", "Synthetic window", _SYNTHETIC_COLOR, "dash")]
    if "training_values" in match:
        windows.insert(0, ("training_values", "Training window", _TRAINING_COLOR, "solid"))
    for column_index, column in enumerate(details["columns"]):
        visible = column_index == 0
        for key, name, color, dash in windows:
            figure.add_trace(
                go.Scatter(
                    x=steps,
                    y=match[key][column_index],
                    mode="lines+markers",
                    name=name,
                    line={"color": color, "dash": dash},
                    hovertemplate=f"{name}<br>{column}<br>step %{{x}}: %{{y:.3g}}<extra></extra>",
                    visible=visible,
                )
            )
    figure.update_layout(
        template="plotly_white",
        height=_FIGURE_HEIGHT,
        showlegend=False,
        xaxis_title="Step in window",
        yaxis_title="Value",
        margin={"l": 54, "r": 16, "t": 40, "b": 48},
    )
    figure.update_xaxes(range=[0.5, details["window_size"] + 0.5])
    return figure