Skip to content

timeseries_preprocessing

timeseries_preprocessing

Time series preprocessing utilities for Safe Synthesizer training.

Functions:

Name Description
process_timeseries_data

Process time series data and validate/infer timestamp parameters.

process_timeseries_data(training_df, config)

Process time series data and validate/infer timestamp parameters.

Normalizes grouped and ungrouped time series into the same training path. When no group column is configured, a reserved pseudo-group column (PSEUDO_GROUP_COLUMN) is added so the whole dataset is treated as one sequence. Timestamp format and interval metadata inferred here are saved back into the resolved config for generation.

This function: 1. Creates a timestamp column if one doesn't exist 2. Validates the timestamp column exists and has no missing values 3. Sorts the data by timestamp 4. Infers timestamp_format from the data 5. Validates or infers timestamp_interval_seconds 6. Sets start_timestamp and stop_timestamp

Parameters:

Name Type Description Default
training_df DataFrame

The training DataFrame.

 required
config SafeSynthesizerParameters

The configuration object with time_series settings

 required

Returns:

Type Description
tuple[DataFrame, SafeSynthesizerParameters]

Tuple of (processed DataFrame, updated config)

Raises:

Type Description
ParameterError

If the timestamp column is missing, if timestamp_format="elapsed_seconds" is set on a non-numeric column, or if an explicit format fails to parse the data.
DataError

If the timestamp column has missing values or intervals are inconsistent.
Source code in src/nemo_safe_synthesizer/training/timeseries_preprocessing.py 
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
def process_timeseries_data(
    training_df: pd.DataFrame,
    config: SafeSynthesizerParameters,
) -> tuple[pd.DataFrame, SafeSynthesizerParameters]:
    """Process time series data and validate/infer timestamp parameters.

    Normalizes grouped and ungrouped time series into the same training path.
    When no group column is configured, a reserved pseudo-group column
    (``PSEUDO_GROUP_COLUMN``) is added so the whole dataset is treated as one
    sequence. Timestamp format and interval metadata inferred here are saved
    back into the resolved config for generation.

    This function:
    1. Creates a timestamp column if one doesn't exist
    2. Validates the timestamp column exists and has no missing values
    3. Sorts the data by timestamp
    4. Infers timestamp_format from the data
    5. Validates or infers timestamp_interval_seconds
    6. Sets start_timestamp and stop_timestamp

    Args:
        training_df: The training DataFrame.
        config: The configuration object with time_series settings

    Returns:
        Tuple of (processed DataFrame, updated config)

    Raises:
        ParameterError: If the timestamp column is missing, if ``timestamp_format="elapsed_seconds"``
            is set on a non-numeric column, or if an explicit format fails to parse the data.
        DataError: If the timestamp column has missing values or intervals are inconsistent.
    """
    ts_config = config.time_series
    original_group_column = config.data.group_training_examples_by
    original_timestamp_column = ts_config.timestamp_column
    validation = validate_timeseries_data(training_df, config)
    training_df = validation.data
    if original_group_column is None:
        logger.info("No group column specified, treating entire dataset as a single sequence")
    if original_timestamp_column is None:
        logger.info(f"Added timestamp column '{validation.timestamp_column}' with elapsed seconds")
    config.data.group_training_examples_by = validation.group_by_column
    config.data.order_training_examples_by = validation.timestamp_column
    ts_config.timestamp_column = validation.timestamp_column
    ts_config.timestamp_format = validation.timestamp_format
    ts_config.timestamp_interval_seconds = validation.timestamp_interval_seconds
    ts_config.start_timestamp = validation.start_timestamp
    ts_config.stop_timestamp = validation.stop_timestamp
    is_elapsed_time = validation.is_elapsed_time
    logger.info(f"Resolved time-series timestamp format: {validation.timestamp_format}")
    if validation.timestamp_interval_seconds is not None:
        logger.info(f"Resolved timestamp_interval_seconds: {validation.timestamp_interval_seconds}s")
    logger.info(
        f"Time series range (consistent across {len(validation.group_stats)} groups): "
        f"{validation.start_timestamp} to {validation.stop_timestamp}",
    )

    # Step 7: Convert timestamp back to string format
    # Skip string conversion for elapsed_seconds format (values are already numeric)
    if (
        not is_elapsed_time
        and ts_config.timestamp_format is not None
        and ts_config.timestamp_format != "elapsed_seconds"
    ):
        training_df[ts_config.timestamp_column] = training_df[ts_config.timestamp_column].dt.strftime(
            ts_config.timestamp_format
        )

    return training_df, config