Skip to content

external_results

external_results

Public result models returned by the Safe Synthesizer pipeline.

Classes:

Name Description
SafeSynthesizerTiming

Wall-clock durations for each pipeline stage.
SafeSynthesizerSummary

Aggregated quality, privacy, and record-count metrics for a pipeline run.

SafeSynthesizerTiming pydantic-model

Bases: NSSBaseModel

Wall-clock durations for each pipeline stage.

Fields:

total_time_sec = None pydantic-field

Total end-to-end pipeline duration in seconds.

pii_replacer_time_sec = None pydantic-field

Time spent on PII replacement.

training_time_sec = None pydantic-field

Time spent on model training.

generation_time_sec = None pydantic-field

Time spent generating synthetic records.

evaluation_time_sec = None pydantic-field

Time spent evaluating synthetic data quality.

log_timing(logger)

Emit all timing fields as a structured table via logger.

Source code in src/nemo_safe_synthesizer/config/external_results.py 
37
38
39
40
41
42
def log_timing(self, logger: logging.Logger) -> None:
    """Emit all timing fields as a structured table via *logger*."""
    logger.info(
        "Safe Synthesizer timing",
        extra={"ctx": {"render_table": True, "tabular_data": self.model_dump(), "title": "Pipeline Timing"}},
    )

log_wandb(run=None)

Update final timing metrics on an active Weights & Biases run.

Parameters:

Name Type Description Default
run Run | None

W&B run instance. No-op when None.

 None
Source code in src/nemo_safe_synthesizer/config/external_results.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
def log_wandb(self, run: wandb.Run | None = None) -> None:
    """Update final timing metrics on an active Weights & Biases run.

    Args:
        run: W&B run instance. No-op when ``None``.
    """
    if run is not None:
        try:
            run.summary.update(
                {
                    "total_time_sec": self.total_time_sec,
                    "pii_replacer_time_sec": self.pii_replacer_time_sec,
                    "training_time_sec": self.training_time_sec,
                    "generation_time_sec": self.generation_time_sec,
                    "evaluation_time_sec": self.evaluation_time_sec,
                }
            )
        except Exception as exc:  # noqa: BLE001 -- observability is best-effort
            logger.runtime.warning("Failed to update W&B timing summary: %s", exc)

SafeSynthesizerSummary pydantic-model

Bases: NSSBaseModel

Aggregated quality, privacy, and record-count metrics for a pipeline run.

Token-field invariants (when all referenced fields are populated):

  • ``num_non_record_tokens == num_completion_tokens
  • num_valid_record_tokens - num_invalid_record_tokens`` (clamped to 0 if slight tokenizer-boundary drift makes the subtraction negative).
  • tokens_per_prompt == num_completion_tokens / num_prompts.
  • valid_record_token_fraction == num_valid_record_tokens / num_completion_tokens.

Fields:

synthetic_data_quality_score = None pydantic-field

Weighted composite of the five sub-scores below (SQS). Higher is better (0--10 scale).

column_correlation_stability_score = None pydantic-field

How closely pairwise column correlations in synthetic data match the original for numeric and categorical columns.

deep_structure_stability_score = None pydantic-field

PCA-based comparison of multivariate structure between real and synthetic data for numeric and categorical columns.

column_distribution_stability_score = None pydantic-field

Per-column Jensen-Shannon distance between training and synthetic distributions averaged across all numeric and categorical columns.

text_semantic_similarity_score = None pydantic-field

Embedding-based semantic closeness between real and synthetic free-text columns.

text_structure_similarity_score = None pydantic-field

Jensen-Shannon divergence over sentence count, words-per-sentence, and characters-per-word distributions between real and synthetic free-text columns.

data_privacy_score = None pydantic-field

Composite of MIA and AIA protection scores.

membership_inference_protection_score = None pydantic-field

Resistance to attacks that try to determine whether a record was in the training set.

attribute_inference_protection_score = None pydantic-field

Resistance to attacks that try to infer sensitive attributes from quasi-identifiers.

num_valid_records = None pydantic-field

Count of synthetic records that passed schema and format validation.

num_invalid_records = None pydantic-field

Count of synthetic records filtered out during validation.

num_prompts = None pydantic-field

Total LLM generation prompts issued.

valid_record_fraction = None pydantic-field

Ratio of valid records: num_valid_records / (num_valid_records + num_invalid_records).

num_completion_tokens = None pydantic-field

Total tokens generated by the LLM across all completions.

num_valid_record_tokens = None pydantic-field

Tokens in records that passed validation.

num_invalid_record_tokens = None pydantic-field

Tokens in records that failed validation.

num_non_record_tokens = None pydantic-field

Tokens not part of any recognized record.

valid_record_token_fraction = None pydantic-field

Fraction of total completion tokens in valid records.

tokens_per_prompt = None pydantic-field

Average completion tokens per prompt: num_completion_tokens / num_prompts.

tokens_per_second = None pydantic-field

Total completion tokens divided by generation wall-clock time.

valid_tokens_per_second = None pydantic-field

Valid record tokens divided by generation wall-clock time.

tokenization_overhead_sec = None pydantic-field

Wall-clock seconds spent on tokenization for statistics tracking.

timing pydantic-field

Per-stage wall-clock durations.

log_summary(logger)

Emit all summary metrics as a structured table via logger.

Source code in src/nemo_safe_synthesizer/config/external_results.py 
169
170
171
172
173
174
def log_summary(self, logger: logging.Logger) -> None:
    """Emit all summary metrics as a structured table via ``logger``."""
    logger.info(
        "Safe Synthesizer Summary",
        extra={"ctx": {"render_table": True, "tabular_data": self.model_dump(), "title": "Quality Metrics"}},
    )

log_wandb()

Update all final summary and timing metrics on the active W&B run.

Source code in src/nemo_safe_synthesizer/config/external_results.py 
213
214
215
216
217
218
219
220
221
222
223
224
def log_wandb(self) -> None:
    """Update all final summary and timing metrics on the active W&B run."""
    import wandb

    if wandb.run is not None:
        # Keep ``None`` values so dashboard comparisons show skipped or
        # uncollected metrics rather than silently retaining stale summary
        # values from an earlier update to a resumed W&B run.
        try:
            wandb.run.summary.update(self._wandb_metrics())
        except Exception as exc:  # noqa: BLE001 -- observability is best-effort
            logger.runtime.warning("Failed to update W&B final summary: %s", exc)