Program Runtime¶
Runtime errors, OOM issues, and configuration problems for NeMo Safe Synthesizer. Sections are organized by pipeline phase. For output quality and evaluation metrics, see Synthetic Data Quality. For environment variables, model caching, offline setup, and inference endpoint settings, see Environment Variables.
Quick Reference¶
| Symptom | Likely Cause | Fix |
|---|---|---|
| Install fails on Python 3.15+ | Outside requires-python upper bound |
Use Python 3.11–3.14 |
| "kernels package not installed" | Optional Kernels Hub backend selected without kernels installed |
Set training.attn_implementation: sdpa |
ConnectionError during startup |
No internet / model not cached | Pre-cache models |
| OOM in training | VRAM exhausted | Reduce batch size, quantize |
| OOM in generation | VRAM exhausted | Verify training cleanup |
| OOM in evaluation | Large dataset + PCA | Reduce columns or disable eval |
| "max_sequences_per_example must be 1" | Incompatible DP config | Configuration Reference -- Differential Privacy |
| "Unable to automatically determine a noise multiplier" | Epsilon too low | Increase epsilon or add records |
| "no valid records" in generation | Underfitting / schema mismatch | See GenerationError |
| Generation appears to hang | Normal for large-context models | See Slow Generation |
| Grouped data trains for only 1 step | Too few training examples | See Grouped Training |
| "exceeds context length" | Records too long | Reduce record size |
| "fraction of invalid records" | Generation quality too low | Lower threshold or retrain |
| Metrics show UNAVAILABLE | Too few records / columns | Ensure >= 200 records |
| Low SQS scores | Underfit or too few records | Review distributions |
| Wrong columns replaced, or PII missed | Discovery misclassified a column | Review and edit the plan |
| "timestamp_column has missing values" | Dirty time series data | Clean NaN/nulls from timestamp column |
timestamp_interval_mismatch error |
Timestamps do not follow timestamp_interval_seconds |
Details |
| Pre-flight validation fails | Dataset or config issue | Pre-flight validation codes |
Enabling Debug Logging¶
Most troubleshooting steps below recommend running with debug logging.
Use the -v flag for debug output, or -vv to also include dependency
logs (vLLM, HuggingFace, etc.):
Debug logging adds:
- Generation diagnostics: EOS token configuration, per-prompt finish reasons and token counts, vLLM progress bars
- Training details: data assembly statistics, example counts
- SamplingParams: the full vLLM parameter object used for generation
You can also set the log level via environment variable:
See Running -- Logging for the full logging configuration reference.
Installation¶
Transformers v5 + vLLM Version Selection¶
uv sync fails with an error mentioning incompatible transformers and
vllm requirements.
Safe Synthesizer pins one vLLM release and the transformers range that
works with it; see the engine extra in pyproject.toml for the current
pair. Keep vLLM's constraints intact so the resolver selects that pairing.
If you've vendored or copied parts of pyproject.toml into another project,
avoid adding a broad transformers>=5.0,<6 override for vLLM. That can erase
vLLM's explicit constraints and allow incompatible Transformers releases.
Slow Tokenizer Warning¶
After upgrading to transformers v5 you may see a log line like:
Loaded slow (Python) tokenizer for
<model>— no Rust backend available.
This means the model's tokenizer has no Rust (tokenizers crate)
implementation and v5 fell back to the SentencePiece/Python backend. Data
prep continues to work but tokenization is ~5–10× slower than the fast
path. Common causes:
- Local cached tokenizer is missing
tokenizer.json— re-download withhuggingface-cli download <model> - Model ships only a SentencePiece vocab (
tokenizer.model) with no Rusttokenizer.json— common on older checkpoints; fast conversion may land upstream. trust_remote_code=Truemodel with a custom slow tokenizer class.
The warning is informational. To suppress it, switch to a model with a
fast tokenizer (most popular models do; check
AutoTokenizer.from_pretrained(model).is_fast).
Unsupported Python Versions¶
Safe Synthesizer supports Python 3.11, 3.12, 3.13, and 3.14. Python 3.15+ is
not supported. Attempting to install on an unsupported interpreter fails when
the installer's generated uv pip command runs.
To fix, create a virtual environment with a supported interpreter:
The project's pyproject.toml enforces requires-python = ">=3.11, <3.15", so
package managers will reject the install on unsupported versions.
Training¶
GPU memory, context length, and backend issues during fine-tuning.
Out of Memory During Training¶
Training OOM errors appear during the "Training" phase with HuggingFace Trainer
stack traces. If you see torch.cuda.OutOfMemoryError:
- Enable 4-bit quantization -- the single largest memory saver. Set
training.quantize_model: trueandtraining.quantization_scheme: bnb-4bit. QLoRA stores the frozen base model in 4-bit NF4 while training LoRA adapters in full precision, cutting model weight memory by ~4x. Quantization reduces precision in the frozen weights; in practice QLoRA typically produces results close to full-precision LoRA, but verify with your evaluation report. On Blackwell hardware,nvfp4andmxfp4schemes offer similar 4-bit footprint with hardware-accelerated matmul — see Quantization schemes - Reduce the context window -- see
Context Length and Record Fitting for
how to lower
training.rope_scaling_factor, truncate records, or simplify grouped examples. Longer sequences require more activation memory even with gradient checkpointing enabled - Verify
training.batch_sizeis1(the default). The effective batch size isbatch_size * gradient_accumulation_steps(default 1 x 8 = 8). Peak memory is set by the forward/backward pass on one micro-batch --gradient_accumulation_stepscontrols how many micro-batches accumulate before each optimizer step but does not affect peak memory - Lower
training.max_vram_fraction(default0.8) to leave headroom for other GPU consumers on the same device
GPU memory during LoRA SFT breaks down into three components:
- Base model weights (dominant) -- ~14 GiB for a 7B model in fp16, ~3.5 GiB in 4-bit. Quantization targets this component
- Activations (proportional to sequence length and batch size) -- self-attention computes an n x n score matrix, so activation memory scales quadratically with sequence length. Gradient checkpointing, which Safe Synthesizer enables by default, reduces this by recomputing activations during the backward pass instead of storing them. Context length and batch size target this component
- LoRA adapter gradients and optimizer states (small) -- typically < 1 GiB for standard LoRA ranks
For deeper coverage, see Methods and tools for efficient training on a single GPU in the HuggingFace documentation.
No GPU Detected¶
If training fails to find a GPU, the HuggingFace backend will attempt to use CPU (extremely slow and not recommended for production training).
To diagnose:
- Verify NVIDIA drivers:
nvidia-smi - Verify PyTorch CUDA build:
python -c "import torch; print(torch.cuda.is_available())" - Ensure you installed the CUDA extras, not the CPU-only package. See Installation for the full command with required index URLs.
Context Length and Record Fitting¶
The effective context window (max_seq_length) is a computed property on
ModelMetadata --
base_max_seq_length * rope_scaling_factor. Every training example must
fit within this window. If it doesn't, data assembly fails with a
GenerationError before training even starts.
When Records Don't Fit¶
Two error messages indicate context-length problems during data assembly:
A single training example (schema prompt + records) exceeds
max_seq_length.
The schema prompt alone is wider than max_seq_length -- typically
because the table has too many columns for the model's context window.
How to Fix¶
- Reduce record size -- shorten text fields, drop unnecessary columns, or simplify the schema.
-
When using
data.group_training_examples_by, all records in the same group must fit in context together, making the limit tighter. Consider reducing the number of records per group.Sizing formula (approximate)
Estimate token budget before adjusting parameters. The /4 divisor is a rough heuristic for BPE tokenizers on JSON content (actual ratios vary by tokenizer and content):
tokens_per_group ≈ (records_per_group × chars_per_record) / 4total ≈ prompt_tokens + tokens_per_group × max_sequences_per_example
Example: 5 records × 200 chars ≈ 250 tokens/group; with a 400-token prompt and 3 groups per example:
400 + 250 × 3 = 1150tokens.See Example Generation -- Sizing for per-mode formulas.
-
If using
TinyLlama/TinyLlama-1.1B-Chat-v1.0, increasetraining.rope_scaling_factorto extend the context window. When set to"auto", it is estimated from dataset token counts using heuristics (4 chars per token for text, 1 token per digit) -- this can underestimate for complex or multilingual data.training.rope_scaling_factoris not applicable when usingHuggingFaceTB/SmolLM3-3B(default model) ormistralai/Mistral-7B-Instruct-v0.3.
Error type clarification
These errors are typed as GenerationError in the codebase even though
they fire during data assembly, not during generation proper. They appear
in the pipeline before any training or generation occurs.
Context-length issues can also surface as OOM during training (the model attempts to process sequences near the limit). See Out of Memory During Training for memory-specific fixes like quantization and batch size reduction.
Grouped Data Produces Very Few Training Examples¶
When using data.group_training_examples_by, the assembler packs multiple
groups into each training example to fill the context window. With
large-context models (e.g. SmolLM3 at 12288 tokens) and small records,
a 200-record dataset with 12 groups can compress into as few as 2 training
examples. Combined with gradient_accumulation_steps=8 (default), this
yields only 1 gradient step -- effectively no training.
The training log confirms this pattern:
Training duration is controlled by training.num_input_records_to_sample,
not epochs. The internal data_fraction is computed as
num_input_records_to_sample / total_training_records. Values above 1.0
duplicate and reshuffle the training examples. To get meaningful training
with grouped data:
- Decrease
max_sequences_per_example-- this is the preferred first step. Fewer groups per example means the assembler produces more training examples, which means more gradient steps per epoch -- without increasing training time per step. Start withmax_sequences_per_example: 1and increase only if quality suffers. (Sequential/time-series mode already enforces one group per example, so this knob only applies to grouped mode.) - If reducing groups per example is not enough, increase
num_input_records_to_sample. Set it toN * dataset_sizefor approximately N passes over the data. - Watch for
Total steps = 1in the Trainer output -- this means the model barely trained
Generation¶
VRAM, invalid records, and early stopping during synthetic data production.
Out of Memory During Generation¶
Generation OOM errors appear during the "Generation" phase with vLLM.
GPU allocation defaults to 80% of available VRAM. Training exposes
training.max_vram_fraction to override this; generation does not yet have
an equivalent config field.
- Ensure no other processes hold GPU memory -- training cleanup should release
it, but verify with
nvidia-smi - If the GPU has less memory than expected, check that the training teardown completed before generation started
Slow Generation with Large-Context Models¶
Generation may appear to hang with large-context models (SmolLM3, Llama 3,
etc.) when using data.group_training_examples_by. Two factors combine
to cause long generation times:
max_tokens scales with context window: each generation prompt is allowed
up to max_seq_length output tokens (12,288 for SmolLM3). If the model
produces long outputs before the stop condition fires, each prompt in the
batch takes proportionally longer. Every 60 seconds a heartbeat line is
logged starting with Generation in progress, plus a short note that new
records only appear after a full batch of prompts finishes. Long
stretches with no new records are normal while generation is still running.
Long-tail batch latency: vLLM processes all prompts in a batch
simultaneously, but llm.generate() blocks until every prompt completes.
With temperature=0.9 (default), each prompt samples a different
generation path. Most paths produce compact output and hit the EOS token
quickly, but a few diverge into longer text before the model emits EOS.
This variance is worse with undertrained models -- instead of reliably
producing <|im_start|> records... <|im_end|>, the model sometimes
wanders into chat-style explanations of unpredictable length.
E.g., in one
observed run with SmolLM3-3B, 96 of 100 prompts finished in 67 seconds,
while the last 4 took an additional 93 seconds -- with a single prompt
consuming 61 seconds and 1618 tokens on its own. The batch cannot return
until the slowest prompt finishes. Run with -v to see vLLM's tqdm
progress bar, which shows per-prompt completion rates and makes the tail
effect visible.
If this stage takes more than 10 minutes, you might need to train the model more or examine the training parameters.
To diagnose, check the batch summary logs first. When vLLM reports why
outputs stopped, the Batch Generation Summary includes aggregate
finish_reasons counts. Run with -v (debug logging) for per-prompt
details:
- Sampling parameters, including EOS token configuration,
max_tokens, and stop conditions - Per-prompt output: token count,
finish_reason, andstop_reason
If outputs show finish_reason=stop, the stop condition is working and
the generation time is real inference time. If outputs show
finish_reason=length, they ran to max_tokens without producing an EOS
token or a complete parseable record -- this can indicate insufficient
training or an overly tight output-token budget (see
Grouped Data Produces Very Few Training Examples).
GenerationError¶
Generation failures during synthetic data production. The two most common:
-
The first batch produced zero valid records. The model may be underfitting or the schema may not match the training data. Increase
training.num_input_records_to_sampleto give the model more context, and check training logs for quality issues. If the batch summary showsfinish_reasonsdominated bylength, generation reachedmax_tokensbefore producing valid records; inspect prompt size, schema size, and grouped/time-series prefill length before treating it as model quality alone. -
Too many invalid records across
generation.patienceconsecutive batches. Consider retraining with more records, adjustingtraining.num_input_records_to_sample, or settinggeneration.structured_generation.enabled=true.
For context-length errors during data assembly ("The number of tokens in an
example exceeds the available context length"), see
Context Length and Record Fitting.
Evaluation¶
Memory and scope issues during quality scoring and report generation.
Out of Memory During Evaluation¶
If evaluation OOMs, reduce the evaluation scope or dataset size:
- For wide datasets, PCA computation in deep structure analysis can OOM.
Reduce the number of columns included in evaluation by lowering
evaluation.sqs_report_columnsor by subsetting the input data. If evaluation is not required for your run, disable it entirely withevaluation.enabled: false. - Histogram binning uses the
doanemethod to reduce memory, but very large datasets may still cause issues. Reduceevaluation.sqs_report_columnsorevaluation.sqs_report_rowsto limit the evaluation scope.
Evaluation and Data Quality
SQS scores, UNAVAILABLE metrics, report limits, and low-quality diagnostics are covered in Synthetic Data Quality.
Configuration¶
Defaults, auto-resolution, and validation errors for pipeline parameters.
Surprising Defaults¶
Several defaults may not match your expectations:
| Parameter | Default | Notes |
|---|---|---|
training.batch_size |
1 |
Effective batch = batch_size x gradient_accumulation_steps (8) |
training.validation_ratio |
0.0 |
No validation split by default |
data.holdout |
0.05 |
5% of records held out for evaluation; capped by data.max_holdout (2000) |
data.random_state |
None |
Auto-generates a random seed -- set this value explicitly if you need reproducibility |
generation.num_records |
1000 |
May be too small for production use |
Auto-Resolved Parameters¶
Many parameters accept "auto" and are resolved at runtime by the
AutoConfigResolver.
See Configuration Reference for the full list.
training.rope_scaling_factor-- auto-estimated from dataset token counts; see Context Length and Record Fitting for details and caveatstraining.num_input_records_to_sample-- derived fromrope_scaling_factor * 25000training.learning_rate-- model-specific default fromModelMetadata: Mistral uses 0.0001, all other supported model families use 0.0005data.max_sequences_per_example-- resolves to1when differential privacy is enabled (required to limit per-example gradient contribution),10otherwise for best performanceprivacy.delta-- computed from record count
Use safe-synthesizer config validate to see how "auto" and default values resolve for
your configuration. Note that some "auto" fields (such as
training.rope_scaling_factor and training.num_input_records_to_sample)
require a dataset to resolve -- they will remain "auto" in the validate
output and only resolve during an actual run:
Common Validation Errors¶
order_training_examples_byrequiresgroup_training_examples_by:-
If you set
data.order_training_examples_bywithout also settingdata.group_training_examples_by, config validation will fail. Ordering only makes sense within groups. group_training_examples_bywith comma-separated column names:-
Setting
data.group_training_examples_by: col1,col2in YAML is parsed as the single string"col1,col2", not as two separate columns. The pipeline will fail with aParameterErrorwhen it tries to find a column literally named"col1,col2"in your data:
Only a single column name is supported. Multi-column grouping is not
currently available. If you need to group by multiple columns, consider
creating a composite column in your data before running the pipeline
(e.g. concatenate patient_id and event_id into a new
patient_event_id column).
- Unsupported file extensions:
-
The
urlparameter accepts.csv,.json,.jsonl,.parquet, and.txtfiles. Other formats raise aValueError. - Incompatible DP settings:
-
If
privacy.dp_enabledistruebutdata.max_sequences_per_exampleis not1, config validation will fail with a clear error message. Set it to"auto"and it will resolve correctly.
Differential Privacy
DP errors and privacy budget troubleshooting are covered in Synthetic Data Quality.
Pre-flight Validation Codes¶
When running with --validate (CLI) or process_data(check_only=True) (SDK),
the following codes may appear. For an overview of what pre-flight validates,
how to interpret the output, and how to use the resolved config, see
Running -- run --validate.
The Check column lists the check name (as emitted in the report and
accepted by disabled_checks). preflight.check_crash is synthesized
by the orchestrator and appears attached to the name of whichever check
raised; treat it as metadata on that check rather than a separate
check of its own.
| Code | Severity | Check | Description |
|---|---|---|---|
torch_missing |
error | gpu.cuda |
PyTorch not installed; cannot verify GPU availability |
no_gpu |
error | gpu.cuda |
No CUDA GPU detected (required for training or generation) |
low_vram |
warning | gpu.vram |
Free GPU VRAM may be insufficient |
vram_exceeds_capacity |
error | gpu.vram |
Estimated training VRAM is far above available GPU memory |
inference_model_missing |
error | env.inference |
NSS_INFERENCE_ENDPOINT is set but no model ID is (replace_pii.llm.model_id, --inference-model-id, or NSS_INFERENCE_MODEL) |
inference_request_settings_invalid |
error | env.inference |
NSS_INFERENCE_TIMEOUT is not a positive number of seconds, or NSS_INFERENCE_REQUEST_OPTIONS is not a JSON object or sets a field Safe Synthesizer manages |
inference_local_profile_invalid |
error | env.inference |
The local vLLM profile cannot be loaded, NSS_INFERENCE_ENDPOINT is not a usable loopback address for it, or a configured model differs from the profile's model |
inference_local_runtime_unavailable |
error | env.inference |
No inference endpoint is set, so Safe Synthesizer would run a local vLLM server, but vLLM, a CUDA GPU, enough GPUs for the profile's tensor_parallel_size, or the profile's min_gpu_memory_gib is unavailable; set NSS_INFERENCE_ENDPOINT to use a remote service |
inference_endpoint_invalid |
error | env.inference |
The configured PII inference endpoint is not an absolute HTTP(S) URL, embeds credentials, or uses plain HTTP for a non-loopback host |
hf_token_missing |
warning | env.hf_model_availability |
Neither HF_TOKEN nor HUGGING_FACE_HUB_TOKEN set, and model loading may need online Hugging Face access |
hf_model_not_cached |
warning/error | env.hf_model_availability |
Hugging Face model is not present in the local cache; severity is error when HF offline mode is enabled |
hf_model_cache_incomplete |
warning/error | env.hf_model_availability |
Cached Hugging Face model snapshot is missing required config, tokenizer, weights, or shards; severity is error when HF offline mode is enabled |
hf_remote_code_not_cached |
warning/error | env.hf_model_availability |
Trusted model references remote code that is not cached locally; severity is error when HF offline mode is enabled |
preflight.check_crash |
error | (crashing check) | A check raised an unexpected exception; the issue's check field names the crashing check and other checks continued running |
column_not_found |
error | columns.groupby / columns.orderby / columns.mia_excluded |
Required column missing from dataset, or input DataFrame uses unsupported MultiIndex columns |
column_nulls |
error | columns.groupby / columns.orderby |
Required column contains null values |
duplicate_columns |
error | timeseries.shape |
Duplicate column names; rename or remove them |
pseudo_column_collision |
error | columns.pseudo |
Dataset contains reserved internal column name, or input DataFrame uses unsupported MultiIndex columns |
constant_column |
warning | columns.constant |
Column has only one unique value |
timestamp_not_found |
error | timeseries.timestamp |
Timestamp column missing, or input DataFrame uses unsupported MultiIndex columns |
timestamp_nulls |
error | timeseries.timestamp |
Timestamp column has nulls |
timestamp_format_mismatch |
error | timeseries.shape |
Timestamp format could not be inferred or the configured format does not match the timestamp values |
timestamp_parse_failed |
error | timeseries.shape |
One or more timestamp values could not be parsed with the inferred or configured timestamp format |
timestamp_elapsed_non_numeric |
error | timeseries.shape |
timestamp_format='elapsed_seconds' was configured for a non-numeric timestamp column |
timestamp_elapsed_invalid |
error | timeseries.shape |
timestamp_format='elapsed_seconds' was configured for boolean or infinite timestamp values |
timestamp_interval_mismatch |
error | timeseries.shape |
Spacing between consecutive timestamps in a group does not match timestamp_interval_seconds |
timeseries_empty |
error | timeseries.shape |
Time-series data contains no records to validate |
tokenizer_unavailable |
warning | token_budget |
Model tokenizer could not be loaded; token checks skipped |
schema_exceeds_context |
error | token_budget |
Schema prompt exceeds model context window |
record_exceeds_context |
error | token_budget |
Individual records exceed context window |
group_exceeds_context |
error | token_budget |
Grouped records exceed context window |
dataset_too_small |
error | dataset.size |
Dataset has fewer than minimum required rows |
dataset_small |
warning | dataset.row_count |
Training set below 1000 records |
extreme_oversampling |
warning | training.oversampling |
Data fraction exceeds 5x |
PII Replacement¶
safe-synthesizer run replace-pii resolves the plan and applies replacement in
one run. Add --plan-only only when you want to validate and persist the
resolved plan without replacing rows. Plan, dependency, and pattern errors are
reported as configuration errors without logging source values.
Free-text targets require the configured GLiNER2 checkpoint. The default is
fastino/gliner2-privacy-filter-PII-multi; allow Hugging Face downloads on the
first run or pre-populate the model cache before using
--disable-huggingface-remote. A model-load or inference failure stops the run
instead of silently continuing with regex detection only.
Automatic Plan Discovery Fails¶
Automatic discovery needs a planning model. By default it runs on a local vLLM
server on your GPU; with NSS_INFERENCE_ENDPOINT set, it uses that service
instead. Common failures:
The local vLLM server exited with code ... before it became ready: the default planning model needs one GPU with 80 GB of memory. Use a GPU that size with no other jobs on it, or use one of the options below. The message ends with the server's last output lines, which usually name the cause, such as running out of GPU memory.Cannot start the local PII inference server on ... because the address is in use: another process is listening on the planning port. Stop it, or follow the message to use that server or choose another port.PII inference authentication or authorization failed: the hosted service rejectedNSS_INFERENCE_KEY.NSS_INFERENCE_ENDPOINT is set, so the model it serves must be set too: setNSS_INFERENCE_MODELorreplace_pii.llm.model_idto the model the service serves.... failed after 3 attempts: requests to the planning model kept failing, for example because the service is unreachable or overloaded. Check the endpoint URL and try again later.... returned invalid structured output after 3 attempts: the model kept giving answers that do not fit the plan format. Use the default planning model or a larger one.
If discovery cannot run where you train, you have two other options:
- Send planning to a hosted model service. Column samples then leave your machine; see Where planning requests go.
- Write the plan yourself, or discover it once on a machine with a large enough GPU and reuse the saved plan. See Customizing PII Replacement.
GLiNER2 Device and Throughput¶
Safe Synthesizer loads GLiNER2 on CUDA. When PyTorch reports no GPU, a run with free-text
columns stops with GLiNER2 free-text PII detection needs a CUDA GPU instead of
falling back to CPU, which is too slow for real datasets. Street-address
columns do not stop the run: without GLiNER2, each full street address is
replaced whole, with a warning, instead of keeping its city and postal code.
Agent sandboxes and containers often hide a GPU the machine has; run Safe Synthesizer where
nvidia-smi lists it. Model loading and inference emit completion records and a progress
heartbeat every 30 seconds; the records include the selected device, chunk
count, configured batch size, and requested-label count without source text.
Free-text inference deduplicates exact cell values, splits each unique value
into overlapping character chunks, and sends the flattened chunks through
GLiNER2's batched API. Start with the defaults (batch_size: 8,
chunk_length: 384, and chunk_overlap: 128). Reduce batch_size to 4, 2, or
1 after a CUDA out-of-memory error. Increase it gradually only when a GPU has
headroom; it does not change chunk boundaries or detection semantics.
The checkpoint favors recall and documents that it can over-predict names.
free_text_detection.entity_thresholds contains one threshold for every
GLiNER2-detected entity. The defaults are tuned on a PII detection benchmark,
and most are above 0.97 because the checkpoint's scores cluster just below 1.0,
so change them in small steps. Email, payment-card, IPv4, and IPv6 detection is
owned by structurally validated regex and has no confidence threshold. If
medical terms, product names, or common nouns are replaced as people, raise the
relevant name entries or add a matching label to
free_text_detection.name_exclusion_thresholds; it is merged into the
default labels. If real names are missed, drop the label that removes them by
setting it to null, such as gender: null: the default exclusion labels also
drop names the model scores as, for example, a gender term. Check both false positives
and missed real names on a representative development sample before and after
any change.
The default checkpoint's documented languages are English, French, Spanish, German, Italian, Portuguese, and Dutch. Natural-language date parsing happens only after GLiNER2 detects a birth-date span, so the parser does not extend the checkpoint's detection-language coverage.
GLiNER2 birth-date candidates must contain a complete parseable day, month,
and year. Natural-language and multilingual complete dates are accepted; vague
phrases such as spring are skipped with a PII-free aggregate warning. If a
complete date is unexpectedly skipped, inspect the aggregate warning count
with -v and report the date format without sharing the original value.
Nemotron-Personas sampling expects <managed-assets>/datasets/<locale>.parquet. If the file is
missing, first download the dataset for replace_pii.locale, as described under
Nemotron-Personas. Set
replace_pii.sampler.nemotron_personas_path or NSS_NEMOTRON_PERSONAS_PATH when the assets are
stored elsewhere. When replace_pii.sampler is not set, Safe Synthesizer uses Faker if the locale file is
missing, unless the plan maps ethnicity values. When replace_pii.sampler names this
sampler, or the plan maps ethnicity values, and the locale file is missing or unreadable, replacement fails
with a configuration error; set replace_pii.sampler.backend: faker and remove any ethnicity
mapping to replace PII without persona data. Fields a persona row lacks, and entity types that are
not persona fields, still use Faker. Dependency values that select no Nemotron-Personas candidates
are configuration errors and must be fixed in data_to_sampler_value_mapping.
If data_to_sampler_value_mapping does not match the ... sampler stops a run, the plan was likely
discovered for another sampler or locale. The error lists every entry the sampler cannot use; change
those entries, or set replace_pii.sampler and replace_pii.locale to the ones the plan was
written for. Gender entries may always use female and male.
WandB¶
Authentication Failures¶
WandB requires an API key when running in online mode. If the key is missing
or invalid, training will fail when the WandB run is initialized.
Set the API key before running:
Or switch to offline mode to avoid network access entirely:
See Running Safe Synthesizer -- WandB Integration for the full WandB setup.
Resume Errors¶
If a WandB run fails to resume (e.g., the run ID no longer exists on the WandB server),
pass --wandb-resume-job-id with a valid run ID from the same WandB project, or
remove the argument to start a fresh WandB run.
Time Series¶
Experimental
Time series synthesis is an experimental feature. APIs and behavior may change between releases.
Time series synthesis has additional validation and generation requirements. For configuration examples, see Configuration -- Time Series.
Common Issues¶
- Missing timestamp values:
-
Any
NaNornullvalues in the timestamp column raise aDataError. Clean your data before running the pipeline:
Interval mismatch¶
timestamp_interval_seconds is an assertion about your data. If the spacing
between consecutive timestamps in any group differs from it, pre-flight and
training fail with timestamp_interval_mismatch. Unset
timestamp_interval_seconds in the config to remove the check, or correct the
input data to have regular intervals by fixing inconsistent timestamp values
or removing the offending groups. Groups may still differ in length, start, or
stop when the interval is set.
- Invalid timestamp data:
-
Null, non-finite, unparseable, or format-incompatible timestamps remain errors. Differently shaped groups are handled automatically; malformed timestamps are not repaired.
- Conflicting timestamp and order columns:
-
In time-series mode,
time_series.timestamp_columnanddata.order_training_examples_byplay the same role. Set either one, or both with the same value; differing values are a configuration error.order_training_examples_byalso requiresdata.group_training_examples_by, so usetimestamp_columnfor a single ungrouped series. - Groups skipped during generation:
-
If a group consistently produces invalid records (exceeding
generation.patienceconsecutive batches abovegeneration.invalid_fraction_threshold), that group is skipped and its rows are discarded. The generation status is thenincomplete. Check your training data quality for those groups. - Out-of-order records:
-
During generation, records are validated for chronological order. Records that arrive out of order are marked invalid.
Groups with different starts or lengths¶
Groups do not need to share a start timestamp, stop timestamp, record count,
or interval. When any of these differ, training preprocessing automatically
selects flexible time-series processing and logs the constraints that
differed. The source timestamp is generated as a synthesized column while
generation validates an internal zero-based sequence index. Generated
timestamps must not decrease within a group and must follow
timestamp_interval_seconds when it is set. Each group stops at a learned
final-row marker, and the maximum source-group length is a hard limit for
synthetic groups.
Error Classes¶
Safe Synthesizer uses a structured error hierarchy. Understanding which error
class you received helps narrow down the cause and write targeted except clauses.
Inheritance:
SafeSynthesizerError
├── InternalError (also RuntimeError)
└── UserError
├── DataError (also ValueError)
├── ParameterError (also ValueError)
└── GenerationError (also RuntimeError)
SDK callers can catch UserError to handle all user-facing errors, or
SafeSynthesizerError to also catch internal errors. Catching the built-in
base (ValueError, RuntimeError) also works since each class inherits from
both.
DataError¶
Bad input data -- NaNs, unsupported types, empty DataFrames, missing values in group or timestamp columns.
Checklist:
- Verify your CSV loads cleanly with
pd.read_csv() - Check for mixed types in columns
- Check that column names in your config match the actual data
- For time series, ensure no nulls in timestamp or group columns
Context-length errors (records too long for the model) raise GenerationError,
not DataError -- see Context Length and Record Fitting.
ParameterError¶
Invalid configuration -- missing columns referenced in config, incompatible option combinations, or missing required parameters. The stacktrace will indicate which parameter is invalid.
Checklist:
- Run
safe-synthesizer config validate --config config.yaml - Verify column names in
group_training_examples_byandorder_training_examples_byexist in your data - For DP, ensure all required privacy parameters are set
GenerationError¶
Errors during generation or data assembly. Two common cases:
- Sampling failures (no valid records, patience exceeded) -- see GenerationError in the Generation section
- Context-length errors during data assembly (records too long for the model) -- see Context Length and Record Fitting
InternalError¶
Library bugs. If you encounter this error through documented interfaces, please file an issue on GitHub.
- Running Safe Synthesizer -- pipeline execution and CLI commands
- Configuration Reference -- parameter tables
- Synthetic Data Quality -- quality and privacy score diagnostics