Skip to content

timeseries_validation

timeseries_validation

Shared time-series validation for preflight and training preprocessing.

Classes:

Name Description
TimeSeriesValidationReason

Domain-specific reason for a time-series validation failure.

TimeSeriesDataValidationError

Data-side time-series validation failure.

TimeSeriesParameterValidationError

Parameter-side time-series validation failure.

TimeSeriesGroupTimestampStats

Statistics collected from a single group's timestamps.

TimeSeriesValidationResult

Validated time-series data and inferred timestamp metadata.

TimeSeriesInspection

Normalized source data and statistics shared by routing and validation.

TimeSeriesRoutingDecision

Automatic selection between deterministic and flexible time-series processing.

Functions:

Name Description
resolve_elapsed_time_column_name

Return an unused generated timestamp column name.

validate_start_stop_consistency

Validate all groups have the same start/stop timestamps.

validate_deterministic_inspection

Validate deterministic shape constraints and return the resolved data.

validate_timeseries_source_data

Validate time-series source data without selecting a representation.

resolve_timeseries_routing

Select the deterministic or flexible time-series representation.

validate_timeseries_data

Validate deterministic time-series data shape and infer timestamp metadata.

TimeSeriesValidationReason

Bases: Enum

Domain-specific reason for a time-series validation failure.

TimeSeriesDataValidationError(reason, message)

Bases: _TimeSeriesValidationError, DataError

Data-side time-series validation failure.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def __init__(self, reason: TimeSeriesValidationReason, message: str) -> None:
    self.reason = reason
    super().__init__(message)

TimeSeriesParameterValidationError(reason, message)

Bases: _TimeSeriesValidationError, ParameterError

Parameter-side time-series validation failure.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def __init__(self, reason: TimeSeriesValidationReason, message: str) -> None:
    self.reason = reason
    super().__init__(message)

TimeSeriesGroupTimestampStats(group_name, start_timestamp, stop_timestamp, interval_seconds, record_count=0) dataclass

Statistics collected from a single group's timestamps.

Attributes:

Name Type Description
group_name Any

Identifier for the time-series group.

start_timestamp Any

First timestamp in the sorted group.

stop_timestamp Any

Last timestamp in the sorted group.

interval_seconds int | None

Seconds between consecutive timestamps, or None when no consistent interval exists.

record_count int

Number of records in the group.

group_name instance-attribute

Identifier for the time-series group.

start_timestamp instance-attribute

First timestamp in the sorted group.

stop_timestamp instance-attribute

Last timestamp in the sorted group.

interval_seconds instance-attribute

Seconds between consecutive timestamps, or None when no consistent interval exists.

record_count = 0 class-attribute instance-attribute

Number of records in the group.

TimeSeriesValidationResult(data, group_by_column, timestamp_column, timestamp_format, is_elapsed_time, timestamp_interval_seconds, start_timestamp, stop_timestamp, group_stats) dataclass

Validated time-series data and inferred timestamp metadata.

Attributes:

Name Type Description
data DataFrame

Validated and timestamp-sorted DataFrame copy.

group_by_column str

Column used to group time-series records.

timestamp_column str

Column used as the timestamp after generated-column resolution.

timestamp_format str

Resolved timestamp format, either "elapsed_seconds" or a strftime format.

is_elapsed_time bool

Whether timestamps are numeric elapsed seconds.

timestamp_interval_seconds int | None

Validated or inferred interval between timestamps, if available.

start_timestamp str

Common start timestamp shared by all groups.

stop_timestamp str

Common stop timestamp shared by all groups.

group_stats tuple[TimeSeriesGroupTimestampStats, ...]

Per-group timestamp statistics used to derive the result.

data instance-attribute

Validated and timestamp-sorted DataFrame copy.

group_by_column instance-attribute

Column used to group time-series records.

timestamp_column instance-attribute

Column used as the timestamp after generated-column resolution.

timestamp_format instance-attribute

Resolved timestamp format, either "elapsed_seconds" or a strftime format.

is_elapsed_time instance-attribute

Whether timestamps are numeric elapsed seconds.

timestamp_interval_seconds instance-attribute

Validated or inferred interval between timestamps, if available.

start_timestamp instance-attribute

Common start timestamp shared by all groups.

stop_timestamp instance-attribute

Common stop timestamp shared by all groups.

group_stats instance-attribute

Per-group timestamp statistics used to derive the result.

TimeSeriesInspection(data, group_column, timestamp_column, timestamp_format, is_elapsed_time, group_stats) dataclass

Normalized source data and statistics shared by routing and validation.

Attributes:

Name Type Description
data DataFrame

Timestamp-sorted source data.

group_column str

Resolved group column in data.

timestamp_column str

Resolved timestamp column in data.

timestamp_format str

Validated or inferred source timestamp format.

is_elapsed_time bool

Whether timestamps are numeric elapsed seconds.

group_stats tuple[TimeSeriesGroupTimestampStats, ...]

Per-group timestamp statistics.

data instance-attribute

Timestamp-sorted source data.

group_column instance-attribute

Resolved group column in data.

The configured group_training_examples_by, or the internal pseudo-group column PSEUDO_GROUP_COLUMN added for an ungrouped series.

timestamp_column instance-attribute

Resolved timestamp column in data.

The configured timestamp_column; the order column when it is used as the timestamp together with timestamp_interval_seconds; or a generated elapsed-seconds column when only timestamp_interval_seconds is set.

timestamp_format instance-attribute

Validated or inferred source timestamp format.

is_elapsed_time instance-attribute

Whether timestamps are numeric elapsed seconds.

group_stats instance-attribute

Per-group timestamp statistics.

TimeSeriesRoutingDecision(use_flexible_timeseries, failed_constraints, sequence_max_records, timestamp_format, inspection, timeseries_metadata=None) dataclass

Automatic selection between deterministic and flexible time-series processing.

Attributes:

Name Type Description
use_flexible_timeseries bool

Whether source shape requires marker-based flexible processing.

failed_constraints tuple[str, ...]

Deterministic shape constraints not satisfied by the source data.

sequence_max_records int

Largest source-group length used as the flexible generation cap.

timestamp_format str

Validated or inferred format used to order source timestamps.

inspection TimeSeriesInspection

Normalized source inspection reused by deterministic preprocessing.

timeseries_metadata TimeseriesMetadata | None

Resolved representation and source schema; flexible controls are set only for flexible routing.

use_flexible_timeseries instance-attribute

Whether source shape requires marker-based flexible processing.

failed_constraints instance-attribute

Deterministic shape constraints not satisfied by the source data.

sequence_max_records instance-attribute

Largest source-group length used as the flexible generation cap.

timestamp_format instance-attribute

Validated or inferred format used to order source timestamps.

inspection instance-attribute

Normalized source inspection reused by deterministic preprocessing.

timeseries_metadata = None class-attribute instance-attribute

Resolved representation and source schema; flexible controls are set only for flexible routing.

resolve_elapsed_time_column_name(columns)

Return an unused generated timestamp column name.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def resolve_elapsed_time_column_name(columns: Iterable[str]) -> str:
    """Return an unused generated timestamp column name."""
    existing = set(columns)
    if "elapsed_seconds" not in existing:
        return "elapsed_seconds"
    if "_elapsed_seconds" not in existing:
        return "_elapsed_seconds"

    suffix = 1
    while True:
        candidate = f"_elapsed_seconds_{suffix}"
        if candidate not in existing:
            return candidate
        suffix += 1

validate_start_stop_consistency(group_stats)

Validate all groups have the same start/stop timestamps.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def validate_start_stop_consistency(
    group_stats: tuple[TimeSeriesGroupTimestampStats, ...] | list[TimeSeriesGroupTimestampStats],
) -> tuple[str, str]:
    """Validate all groups have the same start/stop timestamps."""
    if not group_stats:
        raise TimeSeriesDataValidationError(
            TimeSeriesValidationReason.TIMESERIES_EMPTY,
            "Time-series data must contain at least one record.",
        )

    unique_starts = set(s.start_timestamp for s in group_stats)
    unique_stops = set(s.stop_timestamp for s in group_stats)

    if len(unique_starts) > 1:
        raise TimeSeriesDataValidationError(
            TimeSeriesValidationReason.TIMESERIES_START_MISMATCH,
            f"Start timestamps differ across groups. Found {len(unique_starts)} different start timestamps: "
            f"{sorted([str(t) for t in list(unique_starts)[:5]])}{'...' if len(unique_starts) > 5 else ''}. "
            f"All groups must have the same start timestamp.",
        )

    if len(unique_stops) > 1:
        raise TimeSeriesDataValidationError(
            TimeSeriesValidationReason.TIMESERIES_STOP_MISMATCH,
            f"Stop timestamps differ across groups. Found {len(unique_stops)} different stop timestamps: "
            f"{sorted([str(t) for t in list(unique_stops)[:5]])}{'...' if len(unique_stops) > 5 else ''}. "
            f"All groups must have the same stop timestamp.",
        )

    return str(group_stats[0].start_timestamp), str(group_stats[0].stop_timestamp)

validate_deterministic_inspection(inspection, expected_interval_seconds)

Validate deterministic shape constraints and return the resolved data.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def validate_deterministic_inspection(
    inspection: TimeSeriesInspection,
    expected_interval_seconds: int | None,
) -> TimeSeriesValidationResult:
    """Validate deterministic shape constraints and return the resolved data."""
    _validate_equal_group_lengths(inspection.group_stats)
    start_timestamp, stop_timestamp = validate_start_stop_consistency(inspection.group_stats)
    interval_seconds = _validate_interval_consistency(expected_interval_seconds, inspection.group_stats)
    return TimeSeriesValidationResult(
        data=inspection.data,
        group_by_column=inspection.group_column,
        timestamp_column=inspection.timestamp_column,
        timestamp_format=inspection.timestamp_format,
        is_elapsed_time=inspection.is_elapsed_time,
        timestamp_interval_seconds=interval_seconds,
        start_timestamp=start_timestamp,
        stop_timestamp=stop_timestamp,
        group_stats=inspection.group_stats,
    )

validate_timeseries_source_data(data, config)

Validate time-series source data without selecting a representation.

Reports malformed data and asserted-interval mismatches. Differences in group length, start, stop, or unasserted interval are not errors; they select flexible processing during training preprocessing.

Parameters:

Name Type Description Default
data DataFrame

Source data to validate.

required
config SafeSynthesizerParameters

Parameters defining grouping and timestamp behavior. Not mutated.

required

Raises:

Type Description
TimeSeriesDataValidationError

If data is malformed or does not follow an asserted timestamp_interval_seconds.

TimeSeriesParameterValidationError

If configured columns or formats are incompatible with the data.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def validate_timeseries_source_data(data: pd.DataFrame, config: SafeSynthesizerParameters) -> None:
    """Validate time-series source data without selecting a representation.

    Reports malformed data and asserted-interval mismatches. Differences in
    group length, start, stop, or unasserted interval are not errors; they
    select flexible processing during training preprocessing.

    Args:
        data: Source data to validate.
        config: Parameters defining grouping and timestamp behavior. Not mutated.

    Raises:
        TimeSeriesDataValidationError: If data is malformed or does not follow
            an asserted ``timestamp_interval_seconds``.
        TimeSeriesParameterValidationError: If configured columns or formats
            are incompatible with the data.
    """
    if not config.time_series.is_timeseries:
        return
    _check_duplicate_columns(data)
    _inspect_timeseries_constraints(data, config)

resolve_timeseries_routing(data, config)

Select the deterministic or flexible time-series representation.

Resolves the effective timestamp and order columns and persists an inferred timestamp format on config. The decision carries the time-series metadata for either route; its flexible controls are set only when deterministic shape constraints are not satisfied.

Parameters:

Name Type Description Default
data DataFrame

Source data to inspect.

required
config SafeSynthesizerParameters

Parameters updated with the resolved timestamp columns and format.

required

Returns:

Type Description
TimeSeriesRoutingDecision | None

The routing decision, or None when time-series mode is disabled.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def resolve_timeseries_routing(
    data: pd.DataFrame,
    config: SafeSynthesizerParameters,
) -> TimeSeriesRoutingDecision | None:
    """Select the deterministic or flexible time-series representation.

    Resolves the effective timestamp and order columns and persists an inferred
    timestamp format on ``config``. The decision carries the time-series
    metadata for either route; its flexible controls are set only when
    deterministic shape constraints are not satisfied.

    Args:
        data: Source data to inspect.
        config: Parameters updated with the resolved timestamp columns and format.

    Returns:
        The routing decision, or ``None`` when time-series mode is disabled.
    """
    if not config.time_series.is_timeseries:
        return None
    _check_duplicate_columns(data)

    _resolve_timestamp_order_columns(config)
    ts_config = config.time_series
    decision = _inspect_timeseries_constraints(data, config)
    if ts_config.timestamp_format is None:
        ts_config.timestamp_format = decision.timestamp_format
    flexible = None
    if decision.use_flexible_timeseries:
        flexible = resolve_flexible_timeseries_metadata(
            data,
            config,
            decision.sequence_max_records,
            decision.timestamp_format,
        )
    metadata = TimeseriesMetadata(source_columns=tuple(data.columns), flexible=flexible)
    return replace(decision, timeseries_metadata=metadata)

validate_timeseries_data(data, config)

Validate deterministic time-series data shape and infer timestamp metadata.

The validator performs the same timestamp normalization checks needed by training preprocessing, but operates on copies so preflight can run it without mutating the caller's DataFrame or config.

Parameters:

Name Type Description Default
data DataFrame

Training DataFrame to validate.

required
config SafeSynthesizerParameters

Safe Synthesizer parameters containing time-series settings.

required

Returns:

Type Description
TimeSeriesValidationResult

A validation result with a sorted DataFrame copy and resolved timestamp

TimeSeriesValidationResult

metadata suitable for updating the runtime config.

Raises:

Type Description
TimeSeriesParameterValidationError

If configuration references missing columns or incompatible timestamp formats.

TimeSeriesDataValidationError

If the data violates time-series shape invariants.

Source code in src/nemo_safe_synthesizer/data_processing/timeseries_validation.py
def validate_timeseries_data(data: pd.DataFrame, config: SafeSynthesizerParameters) -> TimeSeriesValidationResult:
    """Validate deterministic time-series data shape and infer timestamp metadata.

    The validator performs the same timestamp normalization checks needed by
    training preprocessing, but operates on copies so preflight can run it
    without mutating the caller's DataFrame or config.

    Args:
        data: Training DataFrame to validate.
        config: Safe Synthesizer parameters containing time-series settings.

    Returns:
        A validation result with a sorted DataFrame copy and resolved timestamp
        metadata suitable for updating the runtime config.

    Raises:
        TimeSeriesParameterValidationError: If configuration references missing
            columns or incompatible timestamp formats.
        TimeSeriesDataValidationError: If the data violates time-series shape
            invariants.
    """
    if data.empty:
        raise TimeSeriesDataValidationError(
            TimeSeriesValidationReason.TIMESERIES_EMPTY,
            "Time-series data must contain at least one record.",
        )

    inspection = _inspect_time_series_data(data, config)
    return validate_deterministic_inspection(
        inspection,
        config.time_series.timestamp_interval_seconds,
    )