timeseries_backend
timeseries_backend
¶
Time-series generation backend with chronological validation.
Classes:
| Name | Description |
|---|---|
ProgressSnapshot |
Snapshot configuration for saving partial generation results at progress milestones. |
RecordPromptState |
Mutable prefix/history state used to build record prompts. |
GroupState |
Mutable state for tracking a single group during parallel generation. |
GroupProcessingResult |
Result of processing a generation batch for a single group. |
TimeseriesBackend |
Time-series aware generator that enforces chronological constraints. |
ProgressSnapshot(label, threshold, path, saved=False)
dataclass
¶
Snapshot configuration for saving partial generation results at progress milestones.
Attributes:
| Name | Type | Description |
|---|---|---|
label |
str
|
Human-readable label for the milestone (e.g. |
threshold |
int
|
Record or group count that triggers this snapshot. |
path |
Path
|
File path where the snapshot CSV will be written. |
saved |
bool
|
Whether this snapshot has already been written to disk. |
label
instance-attribute
¶
Human-readable label for the milestone (e.g. "50").
threshold
instance-attribute
¶
Record or group count that triggers this snapshot.
path
instance-attribute
¶
File path where the snapshot CSV will be written.
saved = field(default=False)
class-attribute
instance-attribute
¶
Whether this snapshot has already been written to disk.
RecordPromptState(prefix, history=list(), using_prefix=True)
dataclass
¶
Mutable prefix/history state used to build record prompts.
Methods:
| Name | Description |
|---|---|
add_history |
Accept records, switch from prefix to history, and clamp the window. |
Attributes:
| Name | Type | Description |
|---|---|---|
prefix |
str
|
Incomplete first record used until the first generated record is accepted. |
history |
list[ParsedRecord]
|
Exact accepted record text used as rolling prompt history. |
using_prefix |
bool
|
Whether generation is still completing the initial record prefix. |
prompt_segments |
str | Sequence[str]
|
Return the prefix or individually encoded history records. |
completion_prefix |
str
|
Return bytes prepended to completions during the prefix phase. |
history_text |
str
|
Return history as newline-terminated training-compatible JSONL. |
prefix
instance-attribute
¶
Incomplete first record used until the first generated record is accepted.
history = field(default_factory=list)
class-attribute
instance-attribute
¶
Exact accepted record text used as rolling prompt history.
using_prefix = True
class-attribute
instance-attribute
¶
Whether generation is still completing the initial record prefix.
prompt_segments
property
¶
Return the prefix or individually encoded history records.
completion_prefix
property
¶
Return bytes prepended to completions during the prefix phase.
history_text
property
¶
Return history as newline-terminated training-compatible JSONL.
add_history(records, *, max_records)
¶
Accept records, switch from prefix to history, and clamp the window.
Source code in src/nemo_safe_synthesizer/generation/timeseries_backend.py
GroupState(group_id, group_ordinal, prompt_state, expected_records=0, last_timestamp_seconds=None, last_source_timestamp_seconds=None, batches=list(), low_valid_fraction_count=0, completed=False, failed=False, total_valid_records=0, total_invalid_records=0, no_progress_count=0, termination_reason=None, total_prompts=0, total_prompt_tokens=0, total_completion_tokens=0)
dataclass
¶
Mutable state for tracking a single group during parallel generation.
Each group maintains its own sliding-window context, timestamp cursor, and retry counters so that multiple groups can be generated in parallel while tracking progress independently.
Attributes:
| Name | Type | Description |
|---|---|---|
group_id |
TimeSeriesGroupValue
|
Unique identifier for this group (e.g., device ID, customer ID). |
group_ordinal |
int
|
Stable one-based position in the saved group registry, used in diagnostics. |
prompt_state |
RecordPromptState
|
Prefix/history container for building prompts and parsing completions. |
expected_records |
int
|
Target record count, calculated from |
last_timestamp_seconds |
int | None
|
Timestamp (in seconds) of the most recently generated record, used for chronological validation. |
last_source_timestamp_seconds |
int | None
|
Flexible mode: source timestamp (in seconds) of the most recently accepted record. |
batches |
list[Batch]
|
Batches whose accepted rows belong to this group; invalidated if the group fails. |
low_valid_fraction_count |
int
|
Consecutive batches with high invalid fraction. Triggers group failure after |
completed |
bool
|
Whether this group has reached the stop timestamp. |
failed |
bool
|
Whether this group failed (e.g., too many retries without progress). |
total_valid_records |
int
|
Cumulative count of valid records generated for this group. |
total_invalid_records |
int
|
Cumulative count of invalid records generated for this group. |
no_progress_count |
int
|
Consecutive batches that did not advance the accepted timestamp. |
termination_reason |
str | None
|
Final flexible time-series outcome: |
total_prompts |
int
|
Number of generation prompts attempted for this group. |
total_prompt_tokens |
int
|
Prompt tokens submitted once per generation request, including retries. |
total_completion_tokens |
int
|
Completion tokens attempted for this group, including discarded rows. |
group_id
instance-attribute
¶
Unique identifier for this group (e.g., device ID, customer ID).
group_ordinal
instance-attribute
¶
Stable one-based position in the saved group registry, used in diagnostics.
prompt_state
instance-attribute
¶
Prefix/history container for building prompts and parsing completions.
expected_records = 0
class-attribute
instance-attribute
¶
Target record count, calculated from (stop_timestamp - start_timestamp) / interval_seconds.
last_timestamp_seconds = None
class-attribute
instance-attribute
¶
Timestamp (in seconds) of the most recently generated record, used for chronological validation.
last_source_timestamp_seconds = None
class-attribute
instance-attribute
¶
Flexible mode: source timestamp (in seconds) of the most recently accepted record.
batches = field(default_factory=list, repr=False)
class-attribute
instance-attribute
¶
Batches whose accepted rows belong to this group; invalidated if the group fails.
low_valid_fraction_count = 0
class-attribute
instance-attribute
¶
Consecutive batches with high invalid fraction. Triggers group failure after patience is exceeded.
completed = False
class-attribute
instance-attribute
¶
Whether this group has reached the stop timestamp.
failed = False
class-attribute
instance-attribute
¶
Whether this group failed (e.g., too many retries without progress).
total_valid_records = 0
class-attribute
instance-attribute
¶
Cumulative count of valid records generated for this group.
total_invalid_records = 0
class-attribute
instance-attribute
¶
Cumulative count of invalid records generated for this group.
no_progress_count = 0
class-attribute
instance-attribute
¶
Consecutive batches that did not advance the accepted timestamp.
termination_reason = None
class-attribute
instance-attribute
¶
Final flexible time-series outcome: terminal, cap, or failure.
total_prompts = 0
class-attribute
instance-attribute
¶
Number of generation prompts attempted for this group.
total_prompt_tokens = 0
class-attribute
instance-attribute
¶
Prompt tokens submitted once per generation request, including retries.
total_completion_tokens = 0
class-attribute
instance-attribute
¶
Completion tokens attempted for this group, including discarded rows.
GroupProcessingResult
¶
Bases: Enum
Result of processing a generation batch for a single group.
Used by _process_group_result to signal whether a group should
remain active, be marked complete, or be removed due to failure.
Attributes:
| Name | Type | Description |
|---|---|---|
IN_PROGRESS |
Group continues; batch should be added to the accumulator. |
|
COMPLETED |
Group reached the stop timestamp; remove from active processing. |
|
FAILED |
Group failed after exhausting a consecutive retry condition. |
IN_PROGRESS = auto()
class-attribute
instance-attribute
¶
Group continues; batch should be added to the accumulator.
COMPLETED = auto()
class-attribute
instance-attribute
¶
Group reached the stop timestamp; remove from active processing.
FAILED = auto()
class-attribute
instance-attribute
¶
Group failed after exhausting a consecutive retry condition.
TimeseriesBackend(config, model_metadata, **kwargs)
¶
Bases: VllmBackend
Time-series aware generator that enforces chronological constraints.
This backend extends VllmBackend to generate synthetic time-series data with strict chronological ordering. It uses a sliding window approach where recently generated records are used as history for subsequent generation, ensuring temporal continuity.
Key Concepts
- Deterministic Time-Range Generation: For fixed-shape training data, the number of records generated is
determined by the configured time range and interval, not by a target
count. Specifically: (stop_timestamp - start_timestamp) / interval_seconds.
The
config.generation.num_recordsparameter is used only for progress tracking, not to limit output. - Flexible Marker Generation: For automatically routed training data, generated rows must have contiguous sequence indices. A final-row marker completes the group, while the largest source-group length provides a safety cap. Internal control columns are removed from final output.
- Sliding Window: The backend maintains a window of recent records
(controlled by
_history_window_size) that are included in each prompt to provide context for the LLM, ensuring generated records follow the established patterns and timestamps. - Parallel Group Generation: Multiple time-series groups (e.g., different
devices, customers) are processed in parallel batches for efficiency.
Even single-sequence data uses this path (treated as 1 group via a
pseudo-group column added during preprocessing). Groups are the same as
those seen during training (from
model_metadata.timeseries_group_values). - Chronological Validation: Each generated record must continue from the previous timestamp at the expected interval. Out-of-order records are marked invalid.
Generation Flow (parallel group mode): 1. Initialize GroupState for each group with a partial first record 2. While groups remain pending or active: a. Fill active slots with pending groups (up to max_groups_per_batch) b. Build prompts for all active groups using their prefix or history c. Generate completions for all prompts in a single LLM batch call d. Process LLM outputs into per-group Batch objects e. For each group: - Validate chronological order against group's last timestamp - Apply data actions and post-processing checks to every sampled response - Retain one validated response (deterministic: most accepted rows; flexible: first ending with the final-row marker, else first valid) - Update group state from accepted records (history, last_timestamp) - Check if an accepted record reached the stop timestamp - Track invalid output and timestamp progress; fail after consecutive retries f. Remove completed/failed groups from active list g. Save progress snapshots if thresholds are met h. Log per-group progress summary
Stopping Conditions
Generation stops when all groups finish (either completed or failed). Individual groups and the overall generation can stop for different reasons:
Per-Group Stopping:
- Completion (success): A deterministic group completes at
_stop_timestamp_value. A flexible group completes at its
final-row marker or source-derived record cap.
- Failure (low valid fraction or no progress): A group fails after
config.generation.patience consecutive batches where either the
invalid record fraction remains above the configured threshold or
no accepted timestamp advances. Failed groups are not retried, and
rows they accepted in earlier batches are discarded from the output.
Global Stopping:
- Natural completion: Generation ends when both the pending groups
queue and active groups list are empty (all groups processed).
- Per-group retry state is authoritative; batch-level stop signals
from GenerationBatches are cleared while groups remain active.
The final status is COMPLETE only when every group completed and
INCOMPLETE when any group failed. num_records does not affect it.
Attributes:
| Name | Type | Description |
|---|---|---|
_samples_per_prompt |
int
|
Minimum number of completion samples per prompt. One sample per group is kept per batch. |
_max_prompts_per_batch |
int
|
Maximum number of prompts to include in a single LLM generation call. Controls parallelism. Default: 100. |
_history_window_size |
int
|
Number of recent records to include in the sliding prompt history. Default: 3. |
_time_column |
str
|
Name of the timestamp column in the data. |
_time_format |
str
|
Format string for parsing timestamps (strptime format), or "elapsed_seconds" for numeric elapsed time. |
_is_elapsed_time |
bool
|
True if timestamps are numeric elapsed seconds. |
_start_timestamp_value |
Starting timestamp for generation range. |
|
_stop_timestamp_value |
Ending timestamp for generation range. Generation stops when a record reaches or exceeds this timestamp. |
|
_timestamp_interval_seconds |
int | None
|
Expected interval between consecutive timestamps. Used for chronological validation. |
_group_column |
str
|
Column name used to group time-series data. |
_group_ordinals |
dict[TimeSeriesGroupValue, int]
|
Mapping of group IDs to stable, non-sensitive positions used in diagnostics. |
_group_prefixes |
dict[TimeSeriesGroupValue, str]
|
Mapping of group IDs to incomplete first records used to start generation. |
_groups |
list[TimeSeriesGroupValue]
|
Typed group IDs to generate. |
_timeseries_metadata |
TimeseriesMetadata | None
|
Resolved representation and source schema from training. |
_flexible_metadata |
FlexibleTimeseriesMetadata | None
|
Resolved
marker-based generation settings; |
Methods:
| Name | Description |
|---|---|
generate |
Generate time-series tabular data using Nemo Safe Synthesizer. |
Source code in src/nemo_safe_synthesizer/generation/timeseries_backend.py
generate(data_actions_fn=None)
¶
Generate time-series tabular data using Nemo Safe Synthesizer.
All time series are processed as groups (single-sequence is treated as 1 group via a pseudo-group column added during preprocessing). Fixed-shape groups stop at the configured time range. Automatically routed flexible groups stop at an accepted final-row marker or the maximum source-group length.
Note
config.generation.num_records is used for progress tracking but
does not limit time-series output or decide completion. The status is
COMPLETE only when every group finishes; otherwise it is
INCOMPLETE and rows from failed groups are discarded. Groups are
the same as those seen during training in
model_metadata.timeseries_group_values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_actions_fn
|
DataActionsFn | None
|
Optional function that takes a DataFrame and returns a modified DataFrame. |
None
|
Returns:
| Type | Description |
|---|---|
GenerateJobResults
|
Generation results object, which includes a DataFrame of generated records. |
Source code in src/nemo_safe_synthesizer/generation/timeseries_backend.py
1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 | |