Skip to content

Program Runtime

Runtime errors, OOM issues, and configuration problems for NeMo Safe Synthesizer. Sections are organized by pipeline phase. For output quality and evaluation metrics, see Synthetic Data Quality. For environment variables, model caching, offline setup, and inference endpoint settings, see Environment Variables.


Quick Reference

Symptom Likely Cause Fix
Install fails on Python 3.15+ Outside requires-python upper bound Use Python 3.11–3.14
"kernels package not installed" Optional Kernels Hub backend selected without kernels installed Set training.attn_implementation: sdpa
ConnectionError during startup No internet / model not cached Pre-cache models
OOM in training VRAM exhausted Reduce batch size, quantize
OOM in generation VRAM exhausted Verify training cleanup
OOM in evaluation Large dataset + PCA Reduce columns or disable eval
"max_sequences_per_example must be 1" Incompatible DP config Configuration Reference -- Differential Privacy
"Unable to automatically determine a noise multiplier" Epsilon too low Increase epsilon or add records
"no valid records" in generation Underfitting / schema mismatch See GenerationError
Generation appears to hang Normal for large-context models See Slow Generation
Grouped data trains for only 1 step Too few training examples See Grouped Training
"exceeds context length" Records too long Reduce record size
"fraction of invalid records" Generation quality too low Lower threshold or retrain
Metrics show UNAVAILABLE Too few records / columns Ensure >= 200 records
Low SQS scores Underfit or too few records Review distributions
Wrong columns replaced, or PII missed Discovery misclassified a column Review and edit the plan
"timestamp_column has missing values" Dirty time series data Clean NaN/nulls from timestamp column
timestamp_interval_mismatch error Timestamps do not follow timestamp_interval_seconds Details
Pre-flight validation fails Dataset or config issue Pre-flight validation codes

Enabling Debug Logging

Most troubleshooting steps below recommend running with debug logging. Use the -v flag for debug output, or -vv to also include dependency logs (vLLM, HuggingFace, etc.):

safe-synthesizer run -v --config config.yaml --url data.csv

Debug logging adds:

  • Generation diagnostics: EOS token configuration, per-prompt finish reasons and token counts, vLLM progress bars
  • Training details: data assembly statistics, example counts
  • SamplingParams: the full vLLM parameter object used for generation

You can also set the log level via environment variable:

export NSS_LOG_LEVEL=DEBUG

See Running -- Logging for the full logging configuration reference.


Installation

Transformers v5 + vLLM Version Selection

uv sync fails with an error mentioning incompatible transformers and vllm requirements.

Safe Synthesizer pins one vLLM release and the transformers range that works with it; see the engine extra in pyproject.toml for the current pair. Keep vLLM's constraints intact so the resolver selects that pairing.

If you've vendored or copied parts of pyproject.toml into another project, avoid adding a broad transformers>=5.0,<6 override for vLLM. That can erase vLLM's explicit constraints and allow incompatible Transformers releases.

Slow Tokenizer Warning

After upgrading to transformers v5 you may see a log line like:

Loaded slow (Python) tokenizer for <model> — no Rust backend available.

This means the model's tokenizer has no Rust (tokenizers crate) implementation and v5 fell back to the SentencePiece/Python backend. Data prep continues to work but tokenization is ~5–10× slower than the fast path. Common causes:

  • Local cached tokenizer is missing tokenizer.json — re-download with huggingface-cli download <model>
  • Model ships only a SentencePiece vocab (tokenizer.model) with no Rust tokenizer.json — common on older checkpoints; fast conversion may land upstream.
  • trust_remote_code=True model with a custom slow tokenizer class.

The warning is informational. To suppress it, switch to a model with a fast tokenizer (most popular models do; check AutoTokenizer.from_pretrained(model).is_fast).

Unsupported Python Versions

Safe Synthesizer supports Python 3.11, 3.12, 3.13, and 3.14. Python 3.15+ is not supported. Attempting to install on an unsupported interpreter fails when the installer's generated uv pip command runs.

To fix, create a virtual environment with a supported interpreter:

uv venv --python 3.13
source .venv/bin/activate

The project's pyproject.toml enforces requires-python = ">=3.11, <3.15", so package managers will reject the install on unsupported versions.


Training

GPU memory, context length, and backend issues during fine-tuning.

Out of Memory During Training

Training OOM errors appear during the "Training" phase with HuggingFace Trainer stack traces. If you see torch.cuda.OutOfMemoryError:

  1. Enable 4-bit quantization -- the single largest memory saver. Set training.quantize_model: true and training.quantization_scheme: bnb-4bit. QLoRA stores the frozen base model in 4-bit NF4 while training LoRA adapters in full precision, cutting model weight memory by ~4x. Quantization reduces precision in the frozen weights; in practice QLoRA typically produces results close to full-precision LoRA, but verify with your evaluation report. On Blackwell hardware, nvfp4 and mxfp4 schemes offer similar 4-bit footprint with hardware-accelerated matmul — see Quantization schemes
  2. Reduce the context window -- see Context Length and Record Fitting for how to lower training.rope_scaling_factor, truncate records, or simplify grouped examples. Longer sequences require more activation memory even with gradient checkpointing enabled
  3. Verify training.batch_size is 1 (the default). The effective batch size is batch_size * gradient_accumulation_steps (default 1 x 8 = 8). Peak memory is set by the forward/backward pass on one micro-batch -- gradient_accumulation_steps controls how many micro-batches accumulate before each optimizer step but does not affect peak memory
  4. Lower training.max_vram_fraction (default 0.8) to leave headroom for other GPU consumers on the same device

GPU memory during LoRA SFT breaks down into three components:

  • Base model weights (dominant) -- ~14 GiB for a 7B model in fp16, ~3.5 GiB in 4-bit. Quantization targets this component
  • Activations (proportional to sequence length and batch size) -- self-attention computes an n x n score matrix, so activation memory scales quadratically with sequence length. Gradient checkpointing, which Safe Synthesizer enables by default, reduces this by recomputing activations during the backward pass instead of storing them. Context length and batch size target this component
  • LoRA adapter gradients and optimizer states (small) -- typically < 1 GiB for standard LoRA ranks

For deeper coverage, see Methods and tools for efficient training on a single GPU in the HuggingFace documentation.

No GPU Detected

If training fails to find a GPU, the HuggingFace backend will attempt to use CPU (extremely slow and not recommended for production training).

To diagnose:

  1. Verify NVIDIA drivers: nvidia-smi
  2. Verify PyTorch CUDA build: python -c "import torch; print(torch.cuda.is_available())"
  3. Ensure you installed the CUDA extras, not the CPU-only package. See Installation for the full command with required index URLs.

Context Length and Record Fitting

The effective context window (max_seq_length) is a computed property on ModelMetadata -- base_max_seq_length * rope_scaling_factor. Every training example must fit within this window. If it doesn't, data assembly fails with a GenerationError before training even starts.

When Records Don't Fit

Two error messages indicate context-length problems during data assembly:

The number of tokens in an example exceeds the available context length

A single training example (schema prompt + records) exceeds max_seq_length.

The dataset schema requires more tokens than the max length of the model

The schema prompt alone is wider than max_seq_length -- typically because the table has too many columns for the model's context window.

How to Fix

  1. Reduce record size -- shorten text fields, drop unnecessary columns, or simplify the schema.
  2. When using data.group_training_examples_by, all records in the same group must fit in context together, making the limit tighter. Consider reducing the number of records per group.

    Sizing formula (approximate)

    Estimate token budget before adjusting parameters. The /4 divisor is a rough heuristic for BPE tokenizers on JSON content (actual ratios vary by tokenizer and content):

    • tokens_per_group ≈ (records_per_group × chars_per_record) / 4
    • total ≈ prompt_tokens + tokens_per_group × max_sequences_per_example

    Example: 5 records × 200 chars ≈ 250 tokens/group; with a 400-token prompt and 3 groups per example: 400 + 250 × 3 = 1150 tokens.

    See Example Generation -- Sizing for per-mode formulas.

  3. If using TinyLlama/TinyLlama-1.1B-Chat-v1.0, increase training.rope_scaling_factor to extend the context window. When set to "auto", it is estimated from dataset token counts using heuristics (4 chars per token for text, 1 token per digit) -- this can underestimate for complex or multilingual data. training.rope_scaling_factor is not applicable when using HuggingFaceTB/SmolLM3-3B (default model) or mistralai/Mistral-7B-Instruct-v0.3.

Error type clarification

These errors are typed as GenerationError in the codebase even though they fire during data assembly, not during generation proper. They appear in the pipeline before any training or generation occurs.

Context-length issues can also surface as OOM during training (the model attempts to process sequences near the limit). See Out of Memory During Training for memory-specific fixes like quantization and batch size reduction.

Grouped Data Produces Very Few Training Examples

When using data.group_training_examples_by, the assembler packs multiple groups into each training example to fill the context window. With large-context models (e.g. SmolLM3 at 12288 tokens) and small records, a 200-record dataset with 12 groups can compress into as few as 2 training examples. Combined with gradient_accumulation_steps=8 (default), this yields only 1 gradient step -- effectively no training.

The training log confirms this pattern:

Num examples = 2 | Num Epochs = 1 | Total steps = 1

Training duration is controlled by training.num_input_records_to_sample, not epochs. The internal data_fraction is computed as num_input_records_to_sample / total_training_records. Values above 1.0 duplicate and reshuffle the training examples. To get meaningful training with grouped data:

  • Decrease max_sequences_per_example -- this is the preferred first step. Fewer groups per example means the assembler produces more training examples, which means more gradient steps per epoch -- without increasing training time per step. Start with max_sequences_per_example: 1 and increase only if quality suffers. (Sequential/time-series mode already enforces one group per example, so this knob only applies to grouped mode.)
  • If reducing groups per example is not enough, increase num_input_records_to_sample. Set it to N * dataset_size for approximately N passes over the data.
  • Watch for Total steps = 1 in the Trainer output -- this means the model barely trained

Generation

VRAM, invalid records, and early stopping during synthetic data production.

Out of Memory During Generation

Generation OOM errors appear during the "Generation" phase with vLLM. GPU allocation defaults to 80% of available VRAM. Training exposes training.max_vram_fraction to override this; generation does not yet have an equivalent config field.

  1. Ensure no other processes hold GPU memory -- training cleanup should release it, but verify with nvidia-smi
  2. If the GPU has less memory than expected, check that the training teardown completed before generation started

Slow Generation with Large-Context Models

Generation may appear to hang with large-context models (SmolLM3, Llama 3, etc.) when using data.group_training_examples_by. Two factors combine to cause long generation times:

max_tokens scales with context window: each generation prompt is allowed up to max_seq_length output tokens (12,288 for SmolLM3). If the model produces long outputs before the stop condition fires, each prompt in the batch takes proportionally longer. Every 60 seconds a heartbeat line is logged starting with Generation in progress, plus a short note that new records only appear after a full batch of prompts finishes. Long stretches with no new records are normal while generation is still running.

Long-tail batch latency: vLLM processes all prompts in a batch simultaneously, but llm.generate() blocks until every prompt completes. With temperature=0.9 (default), each prompt samples a different generation path. Most paths produce compact output and hit the EOS token quickly, but a few diverge into longer text before the model emits EOS. This variance is worse with undertrained models -- instead of reliably producing <|im_start|> records... <|im_end|>, the model sometimes wanders into chat-style explanations of unpredictable length.

E.g., in one observed run with SmolLM3-3B, 96 of 100 prompts finished in 67 seconds, while the last 4 took an additional 93 seconds -- with a single prompt consuming 61 seconds and 1618 tokens on its own. The batch cannot return until the slowest prompt finishes. Run with -v to see vLLM's tqdm progress bar, which shows per-prompt completion rates and makes the tail effect visible.

If this stage takes more than 10 minutes, you might need to train the model more or examine the training parameters.

To diagnose, check the batch summary logs first. When vLLM reports why outputs stopped, the Batch Generation Summary includes aggregate finish_reasons counts. Run with -v (debug logging) for per-prompt details:

  • Sampling parameters, including EOS token configuration, max_tokens, and stop conditions
  • Per-prompt output: token count, finish_reason, and stop_reason

If outputs show finish_reason=stop, the stop condition is working and the generation time is real inference time. If outputs show finish_reason=length, they ran to max_tokens without producing an EOS token or a complete parseable record -- this can indicate insufficient training or an overly tight output-token budget (see Grouped Data Produces Very Few Training Examples).

GenerationError

Generation failures during synthetic data production. The two most common:

Generation stopped prematurely due to no valid records

The first batch produced zero valid records. The model may be underfitting or the schema may not match the training data. Increase training.num_input_records_to_sample to give the model more context, and check training logs for quality issues. If the batch summary shows finish_reasons dominated by length, generation reached max_tokens before producing valid records; inspect prompt size, schema size, and grouped/time-series prefill length before treating it as model quality alone.

Generation stopped prematurely because the average fraction of invalid records was higher than...

Too many invalid records across generation.patience consecutive batches. Consider retraining with more records, adjusting training.num_input_records_to_sample, or setting generation.structured_generation.enabled=true.

For context-length errors during data assembly ("The number of tokens in an example exceeds the available context length"), see Context Length and Record Fitting.


Evaluation

Memory and scope issues during quality scoring and report generation.

Out of Memory During Evaluation

If evaluation OOMs, reduce the evaluation scope or dataset size:

  1. For wide datasets, PCA computation in deep structure analysis can OOM. Reduce the number of columns included in evaluation by lowering evaluation.sqs_report_columns or by subsetting the input data. If evaluation is not required for your run, disable it entirely with evaluation.enabled: false.
  2. Histogram binning uses the doane method to reduce memory, but very large datasets may still cause issues. Reduce evaluation.sqs_report_columns or evaluation.sqs_report_rows to limit the evaluation scope.

Evaluation and Data Quality

SQS scores, UNAVAILABLE metrics, report limits, and low-quality diagnostics are covered in Synthetic Data Quality.


Configuration

Defaults, auto-resolution, and validation errors for pipeline parameters.

Surprising Defaults

Several defaults may not match your expectations:

Parameter Default Notes
training.batch_size 1 Effective batch = batch_size x gradient_accumulation_steps (8)
training.validation_ratio 0.0 No validation split by default
data.holdout 0.05 5% of records held out for evaluation; capped by data.max_holdout (2000)
data.random_state None Auto-generates a random seed -- set this value explicitly if you need reproducibility
generation.num_records 1000 May be too small for production use

Auto-Resolved Parameters

Many parameters accept "auto" and are resolved at runtime by the AutoConfigResolver. See Configuration Reference for the full list.

  • training.rope_scaling_factor -- auto-estimated from dataset token counts; see Context Length and Record Fitting for details and caveats
  • training.num_input_records_to_sample -- derived from rope_scaling_factor * 25000
  • training.learning_rate -- model-specific default from ModelMetadata: Mistral uses 0.0001, all other supported model families use 0.0005
  • data.max_sequences_per_example -- resolves to 1 when differential privacy is enabled (required to limit per-example gradient contribution), 10 otherwise for best performance
  • privacy.delta -- computed from record count

Use safe-synthesizer config validate to see how "auto" and default values resolve for your configuration. Note that some "auto" fields (such as training.rope_scaling_factor and training.num_input_records_to_sample) require a dataset to resolve -- they will remain "auto" in the validate output and only resolve during an actual run:

safe-synthesizer config validate --config config.yaml

Common Validation Errors

order_training_examples_by requires group_training_examples_by:

If you set data.order_training_examples_by without also setting data.group_training_examples_by, config validation will fail. Ordering only makes sense within groups.

group_training_examples_by with comma-separated column names:

Setting data.group_training_examples_by: col1,col2 in YAML is parsed as the single string "col1,col2", not as two separate columns. The pipeline will fail with a ParameterError when it tries to find a column literally named "col1,col2" in your data:

ParameterError: Group by column 'patient_id,event_id' not found in the input data.
The column name contains a comma -- multi-column grouping is not supported.
Use a single column name.

Only a single column name is supported. Multi-column grouping is not currently available. If you need to group by multiple columns, consider creating a composite column in your data before running the pipeline (e.g. concatenate patient_id and event_id into a new patient_event_id column).

Unsupported file extensions:

The url parameter accepts .csv, .json, .jsonl, .parquet, and .txt files. Other formats raise a ValueError.

Incompatible DP settings:

If privacy.dp_enabled is true but data.max_sequences_per_example is not 1, config validation will fail with a clear error message. Set it to "auto" and it will resolve correctly.

Differential Privacy

DP errors and privacy budget troubleshooting are covered in Synthetic Data Quality.

Pre-flight Validation Codes

When running with --validate (CLI) or process_data(check_only=True) (SDK), the following codes may appear. For an overview of what pre-flight validates, how to interpret the output, and how to use the resolved config, see Running -- run --validate.

The Check column lists the check name (as emitted in the report and accepted by disabled_checks). preflight.check_crash is synthesized by the orchestrator and appears attached to the name of whichever check raised; treat it as metadata on that check rather than a separate check of its own.

Code Severity Check Description
torch_missing error gpu.cuda PyTorch not installed; cannot verify GPU availability
no_gpu error gpu.cuda No CUDA GPU detected (required for training or generation)
low_vram warning gpu.vram Free GPU VRAM may be insufficient
vram_exceeds_capacity error gpu.vram Estimated training VRAM is far above available GPU memory
inference_model_missing error env.inference NSS_INFERENCE_ENDPOINT is set but no model ID is (replace_pii.llm.model_id, --inference-model-id, or NSS_INFERENCE_MODEL)
inference_request_settings_invalid error env.inference NSS_INFERENCE_TIMEOUT is not a positive number of seconds, or NSS_INFERENCE_REQUEST_OPTIONS is not a JSON object or sets a field Safe Synthesizer manages
inference_local_profile_invalid error env.inference The local vLLM profile cannot be loaded, NSS_INFERENCE_ENDPOINT is not a usable loopback address for it, or a configured model differs from the profile's model
inference_local_runtime_unavailable error env.inference No inference endpoint is set, so Safe Synthesizer would run a local vLLM server, but vLLM, a CUDA GPU, enough GPUs for the profile's tensor_parallel_size, or the profile's min_gpu_memory_gib is unavailable; set NSS_INFERENCE_ENDPOINT to use a remote service
inference_endpoint_invalid error env.inference The configured PII inference endpoint is not an absolute HTTP(S) URL, embeds credentials, or uses plain HTTP for a non-loopback host
hf_token_missing warning env.hf_model_availability Neither HF_TOKEN nor HUGGING_FACE_HUB_TOKEN set, and model loading may need online Hugging Face access
hf_model_not_cached warning/error env.hf_model_availability Hugging Face model is not present in the local cache; severity is error when HF offline mode is enabled
hf_model_cache_incomplete warning/error env.hf_model_availability Cached Hugging Face model snapshot is missing required config, tokenizer, weights, or shards; severity is error when HF offline mode is enabled
hf_remote_code_not_cached warning/error env.hf_model_availability Trusted model references remote code that is not cached locally; severity is error when HF offline mode is enabled
preflight.check_crash error (crashing check) A check raised an unexpected exception; the issue's check field names the crashing check and other checks continued running
column_not_found error columns.groupby / columns.orderby / columns.mia_excluded Required column missing from dataset, or input DataFrame uses unsupported MultiIndex columns
column_nulls error columns.groupby / columns.orderby Required column contains null values
duplicate_columns error timeseries.shape Duplicate column names; rename or remove them
pseudo_column_collision error columns.pseudo Dataset contains reserved internal column name, or input DataFrame uses unsupported MultiIndex columns
constant_column warning columns.constant Column has only one unique value
timestamp_not_found error timeseries.timestamp Timestamp column missing, or input DataFrame uses unsupported MultiIndex columns
timestamp_nulls error timeseries.timestamp Timestamp column has nulls
timestamp_format_mismatch error timeseries.shape Timestamp format could not be inferred or the configured format does not match the timestamp values
timestamp_parse_failed error timeseries.shape One or more timestamp values could not be parsed with the inferred or configured timestamp format
timestamp_elapsed_non_numeric error timeseries.shape timestamp_format='elapsed_seconds' was configured for a non-numeric timestamp column
timestamp_elapsed_invalid error timeseries.shape timestamp_format='elapsed_seconds' was configured for boolean or infinite timestamp values
timestamp_interval_mismatch error timeseries.shape Spacing between consecutive timestamps in a group does not match timestamp_interval_seconds
timeseries_empty error timeseries.shape Time-series data contains no records to validate
tokenizer_unavailable warning token_budget Model tokenizer could not be loaded; token checks skipped
schema_exceeds_context error token_budget Schema prompt exceeds model context window
record_exceeds_context error token_budget Individual records exceed context window
group_exceeds_context error token_budget Grouped records exceed context window
dataset_too_small error dataset.size Dataset has fewer than minimum required rows
dataset_small warning dataset.row_count Training set below 1000 records
extreme_oversampling warning training.oversampling Data fraction exceeds 5x

PII Replacement

safe-synthesizer run replace-pii resolves the plan and applies replacement in one run. Add --plan-only only when you want to validate and persist the resolved plan without replacing rows. Plan, dependency, and pattern errors are reported as configuration errors without logging source values.

Free-text targets require the configured GLiNER2 checkpoint. The default is fastino/gliner2-privacy-filter-PII-multi; allow Hugging Face downloads on the first run or pre-populate the model cache before using --disable-huggingface-remote. A model-load or inference failure stops the run instead of silently continuing with regex detection only.

Automatic Plan Discovery Fails

Automatic discovery needs a planning model. By default it runs on a local vLLM server on your GPU; with NSS_INFERENCE_ENDPOINT set, it uses that service instead. Common failures:

  • The local vLLM server exited with code ... before it became ready: the default planning model needs one GPU with 80 GB of memory. Use a GPU that size with no other jobs on it, or use one of the options below. The message ends with the server's last output lines, which usually name the cause, such as running out of GPU memory.
  • Cannot start the local PII inference server on ... because the address is in use: another process is listening on the planning port. Stop it, or follow the message to use that server or choose another port.
  • PII inference authentication or authorization failed: the hosted service rejected NSS_INFERENCE_KEY.
  • NSS_INFERENCE_ENDPOINT is set, so the model it serves must be set too: set NSS_INFERENCE_MODEL or replace_pii.llm.model_id to the model the service serves.
  • ... failed after 3 attempts: requests to the planning model kept failing, for example because the service is unreachable or overloaded. Check the endpoint URL and try again later.
  • ... returned invalid structured output after 3 attempts: the model kept giving answers that do not fit the plan format. Use the default planning model or a larger one.

If discovery cannot run where you train, you have two other options:

GLiNER2 Device and Throughput

Safe Synthesizer loads GLiNER2 on CUDA. When PyTorch reports no GPU, a run with free-text columns stops with GLiNER2 free-text PII detection needs a CUDA GPU instead of falling back to CPU, which is too slow for real datasets. Street-address columns do not stop the run: without GLiNER2, each full street address is replaced whole, with a warning, instead of keeping its city and postal code. Agent sandboxes and containers often hide a GPU the machine has; run Safe Synthesizer where nvidia-smi lists it. Model loading and inference emit completion records and a progress heartbeat every 30 seconds; the records include the selected device, chunk count, configured batch size, and requested-label count without source text.

Free-text inference deduplicates exact cell values, splits each unique value into overlapping character chunks, and sends the flattened chunks through GLiNER2's batched API. Start with the defaults (batch_size: 8, chunk_length: 384, and chunk_overlap: 128). Reduce batch_size to 4, 2, or 1 after a CUDA out-of-memory error. Increase it gradually only when a GPU has headroom; it does not change chunk boundaries or detection semantics.

The checkpoint favors recall and documents that it can over-predict names. free_text_detection.entity_thresholds contains one threshold for every GLiNER2-detected entity. The defaults are tuned on a PII detection benchmark, and most are above 0.97 because the checkpoint's scores cluster just below 1.0, so change them in small steps. Email, payment-card, IPv4, and IPv6 detection is owned by structurally validated regex and has no confidence threshold. If medical terms, product names, or common nouns are replaced as people, raise the relevant name entries or add a matching label to free_text_detection.name_exclusion_thresholds; it is merged into the default labels. If real names are missed, drop the label that removes them by setting it to null, such as gender: null: the default exclusion labels also drop names the model scores as, for example, a gender term. Check both false positives and missed real names on a representative development sample before and after any change.

The default checkpoint's documented languages are English, French, Spanish, German, Italian, Portuguese, and Dutch. Natural-language date parsing happens only after GLiNER2 detects a birth-date span, so the parser does not extend the checkpoint's detection-language coverage.

GLiNER2 birth-date candidates must contain a complete parseable day, month, and year. Natural-language and multilingual complete dates are accepted; vague phrases such as spring are skipped with a PII-free aggregate warning. If a complete date is unexpectedly skipped, inspect the aggregate warning count with -v and report the date format without sharing the original value.

Nemotron-Personas sampling expects <managed-assets>/datasets/<locale>.parquet. If the file is missing, first download the dataset for replace_pii.locale, as described under Nemotron-Personas. Set replace_pii.sampler.nemotron_personas_path or NSS_NEMOTRON_PERSONAS_PATH when the assets are stored elsewhere. When replace_pii.sampler is not set, Safe Synthesizer uses Faker if the locale file is missing, unless the plan maps ethnicity values. When replace_pii.sampler names this sampler, or the plan maps ethnicity values, and the locale file is missing or unreadable, replacement fails with a configuration error; set replace_pii.sampler.backend: faker and remove any ethnicity mapping to replace PII without persona data. Fields a persona row lacks, and entity types that are not persona fields, still use Faker. Dependency values that select no Nemotron-Personas candidates are configuration errors and must be fixed in data_to_sampler_value_mapping.

If data_to_sampler_value_mapping does not match the ... sampler stops a run, the plan was likely discovered for another sampler or locale. The error lists every entry the sampler cannot use; change those entries, or set replace_pii.sampler and replace_pii.locale to the ones the plan was written for. Gender entries may always use female and male.


WandB

Authentication Failures

WandB requires an API key when running in online mode. If the key is missing or invalid, training will fail when the WandB run is initialized.

wandb: ERROR api_key not configured (no-auth)

Set the API key before running:

export WANDB_API_KEY="your-api-key"  # pragma: allowlist secret

Or switch to offline mode to avoid network access entirely:

safe-synthesizer run --wandb-mode disabled --config config.yaml --data-source data.csv

See Running Safe Synthesizer -- WandB Integration for the full WandB setup.

Resume Errors

If a WandB run fails to resume (e.g., the run ID no longer exists on the WandB server), pass --wandb-resume-job-id with a valid run ID from the same WandB project, or remove the argument to start a fresh WandB run.


Time Series

Experimental

Time series synthesis is an experimental feature. APIs and behavior may change between releases.

Time series synthesis has additional validation and generation requirements. For configuration examples, see Configuration -- Time Series.

Common Issues

Missing timestamp values:

Any NaN or null values in the timestamp column raise a DataError. Clean your data before running the pipeline:

df = df.dropna(subset=["timestamp"])
df = df.sort_values(by=["group_column", "timestamp"])

Interval mismatch

timestamp_interval_seconds is an assertion about your data. If the spacing between consecutive timestamps in any group differs from it, pre-flight and training fail with timestamp_interval_mismatch. Unset timestamp_interval_seconds in the config to remove the check, or correct the input data to have regular intervals by fixing inconsistent timestamp values or removing the offending groups. Groups may still differ in length, start, or stop when the interval is set.

Invalid timestamp data:

Null, non-finite, unparseable, or format-incompatible timestamps remain errors. Differently shaped groups are handled automatically; malformed timestamps are not repaired.

Conflicting timestamp and order columns:

In time-series mode, time_series.timestamp_column and data.order_training_examples_by play the same role. Set either one, or both with the same value; differing values are a configuration error. order_training_examples_by also requires data.group_training_examples_by, so use timestamp_column for a single ungrouped series.

Groups skipped during generation:

If a group consistently produces invalid records (exceeding generation.patience consecutive batches above generation.invalid_fraction_threshold), that group is skipped and its rows are discarded. The generation status is then incomplete. Check your training data quality for those groups.

Out-of-order records:

During generation, records are validated for chronological order. Records that arrive out of order are marked invalid.

Groups with different starts or lengths

Groups do not need to share a start timestamp, stop timestamp, record count, or interval. When any of these differ, training preprocessing automatically selects flexible time-series processing and logs the constraints that differed. The source timestamp is generated as a synthesized column while generation validates an internal zero-based sequence index. Generated timestamps must not decrease within a group and must follow timestamp_interval_seconds when it is set. Each group stops at a learned final-row marker, and the maximum source-group length is a hard limit for synthetic groups.


Error Classes

Safe Synthesizer uses a structured error hierarchy. Understanding which error class you received helps narrow down the cause and write targeted except clauses.

Inheritance:

SafeSynthesizerError
├── InternalError (also RuntimeError)
└── UserError
    ├── DataError (also ValueError)
    ├── ParameterError (also ValueError)
    └── GenerationError (also RuntimeError)

SDK callers can catch UserError to handle all user-facing errors, or SafeSynthesizerError to also catch internal errors. Catching the built-in base (ValueError, RuntimeError) also works since each class inherits from both.

DataError

Bad input data -- NaNs, unsupported types, empty DataFrames, missing values in group or timestamp columns.

Checklist:

  1. Verify your CSV loads cleanly with pd.read_csv()
  2. Check for mixed types in columns
  3. Check that column names in your config match the actual data
  4. For time series, ensure no nulls in timestamp or group columns

Context-length errors (records too long for the model) raise GenerationError, not DataError -- see Context Length and Record Fitting.

ParameterError

Invalid configuration -- missing columns referenced in config, incompatible option combinations, or missing required parameters. The stacktrace will indicate which parameter is invalid.

Checklist:

  1. Run safe-synthesizer config validate --config config.yaml
  2. Verify column names in group_training_examples_by and order_training_examples_by exist in your data
  3. For DP, ensure all required privacy parameters are set

GenerationError

Errors during generation or data assembly. Two common cases:

InternalError

Library bugs. If you encounter this error through documented interfaces, please file an issue on GitHub.