Time Series - Standardizing#

Real time series data doesn’t always land on a clean, evenly-spaced grid: a sensor logger can skip a reading, drift by a few minutes, or simply have missing rows rather than explicit nulls. The .standardize() method forces the series onto a regular grid at its detected frequency – inserting null rows wherever a timestamp is missing from that grid.

This is a foundational step: many other methods (.get_epochs(), .interpolate_gaps(), .scale_up()) assume a regular grid and call .standardize() internally if the series isn’t already standard. Understanding what .standardize() does – and where it can surprise you – makes those methods much easier to reason about.

Covered here:

What “irregular” data looks like (missing rows on an otherwise regular grid)
Frequency detection (.dtfreq) and the .is_standard flag
What .standardize() actually does to row count and null count
A caveat: how minute-level jitter can silently change the detected frequency and distort the result

Notebook setup#

For users running this tutorial as a Jupyter Notebook, this cell must be executed first:

import sys
from pathlib import Path
import pprint
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Install `plans` in `google.colab`.
# Use `pip install plans` for other environments.

if "google.colab" in sys.modules:
    import os
    os.system(f"{sys.executable} -m pip install -q plans")

# This avoids warnings related to uninstalled fonts
import logging
logging.getLogger('matplotlib.font_manager').setLevel(logging.ERROR)

# define output folder
OUTPUT_DIR = Path("outputs/time-series")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
print(f"Outputs will be saved to: ./{OUTPUT_DIR}")

# fixed seed so the irregular timestamps are reproducible
RNG_SEED = 11
np.random.seed(RNG_SEED)

Outputs will be saved to: ./outputs/time-series

Create a perfect synthetic series#

Same starting point as the other tutorials: a Trend-Seasonality-Noise archetype series, hourly resolution, two weeks of data.

from plans.datasets import TimeSeries

df_perfect = TimeSeries.make_synthetic_tsn(
    start="2020-01-01",
    end="2020-01-15",
    base=100,
    freq="1h",
    trend=0.001,
    noise_sd=4.0,
    amplitude=50,
    seasonal_period="YS",
    minor_amplitude=20,
    minor_seasonal_period="D"
)
print(f"Rows: {len(df_perfect)}")
df_perfect.head()

Rows: 337

	datetime	level
0	2020-01-01 00:00:00	106.997819
1	2020-01-01 01:00:00	104.068854
2	2020-01-01 02:00:00	108.135269
3	2020-01-01 03:00:00	103.639156
4	2020-01-01 04:00:00	117.434429

Make the grid irregular#

This is different from the gaps tutorial: there, values were set to NaN but every timestamp still existed as a row. Here, we drop rows entirely – the timestamps themselves go missing, so the grid is no longer evenly spaced. This mimics a logger that occasionally fails to write a record at all, rather than writing a bad value.

n = len(df_perfect)

# -- drop ~8% of rows entirely (scattered missing timestamps) --
DROP_FRACTION = 0.08
drop_mask = np.random.rand(n) < DROP_FRACTION
df_irregular = df_perfect[~drop_mask].reset_index(drop=True)

print(f"Original rows: {n}")
print(f"Rows after dropping: {len(df_irregular)} "
      f"({n - len(df_irregular)} timestamps missing entirely)")

Original rows: 337
Rows after dropping: 311 (26 timestamps missing entirely)

A quick check on the timestamp deltas confirms the grid is no longer uniform – most gaps are 1 hour, but some are 2 hours or more where a row was dropped:

deltas = df_irregular["datetime"].diff().dropna()
print(deltas.value_counts().sort_index())

datetime
0 days 01:00:00    287
0 days 02:00:00     20
0 days 03:00:00      3
Name: count, dtype: int64

Save and load through the `TimeSeries` object#

Same workflow as the other tutorials: export to CSV, then load through .load_data().

file_csv = OUTPUT_DIR / "time_series_irregular.csv"
df_irregular.to_csv(file_csv, sep=";", index=False)
print(f"Saved to: {file_csv}")

ts = TimeSeries(name="Irregular", alias="irr")
ts.load_data(
    file_data=file_csv,
    input_dtfield="datetime",
    input_varfield="level",
    in_sep=";",
)

Saved to: outputs/time-series/time_series_irregular.csv

Inspect before standardizing#

Two attributes matter here:

.dtfreq – the frequency plans detected from the timestamps (e.g. "h" for hourly). This is inferred from how much variation exists in the seconds/minutes/hours/day components of the data – see the caveat section below for how this can go wrong.
.is_standard – False until .standardize() has been called; tells you whether the series currently sits on a regular grid.

v = ts.varfield
print(f"Rows loaded: {len(ts.data)}")
print(f"Detected frequency (.dtfreq): {ts.dtfreq}")
print(f"Is standard (.is_standard): {ts.is_standard}")
print(f"Null records: {ts.data[v].isna().sum()}")

Rows loaded: 311
Detected frequency (.dtfreq): h
Is standard (.is_standard): False
Null records: 0

Note that null count is zero – the irregularity here is entirely about missing rows, not missing values. .view() on ungapped-but-irregular data won’t visually show anything obviously wrong; the problem only becomes apparent once you look at the timestamp spacing (as above) or once resampling/aggregation methods run into the uneven grid.

Standardize#

.standardize() builds a full, regular date range at .dtfreq spanning the series’ start to end, and left-merges the existing data onto it. Any regular-grid slot with no matching data becomes a null row. It updates self.data in place and sets .is_standard = True.

rows_before = len(ts.data)
ts.standardize()
rows_after = len(ts.data)

print(f"Rows before standardize: {rows_before}")
print(f"Rows after standardize:  {rows_after}")
print(f"Null records introduced: {ts.data[v].isna().sum()}")
print(f"Is standard now: {ts.is_standard}")

Rows before standardize: 311
Rows after standardize:  361
Null records introduced: 50
Is standard now: True

The row count now matches a full regular hourly range, and every missing timestamp from the original irregular data has become an explicit NaN row – exactly the kind of gap the earlier gaps/epochs tutorial works with. Confirming the grid is now evenly spaced:

deltas_std = ts.data[ts.dtfield].diff().dropna()
print(deltas_std.value_counts())

datetime
0 days 01:00:00    360
Name: count, dtype: int64

Robustness check: minute-level jitter#

Frequency detection (.dtfreq) is inferred from timestamp spacing, and standardize()’s bucketing floors each row independently onto that frequency’s grid. Both of these matter for irregular data: a naive detector based on “do the minutes vary at all” would be thrown off by a single jittered timestamp, and naive bucketing based on hardcoded string patterns could accidentally merge rows that should stay in separate slots.

The next cell stress-tests this on a small, isolated example – three timestamps out of 49 nudged by a random number of minutes, everything else exactly hourly:

df_jitter = TimeSeries.make_synthetic_tsn(
    start="2020-01-01",
    end="2020-01-03",
    base=100,
    freq="1h",
    trend=0.0,
    noise_sd=2.0,
    amplitude=10,
    seasonal_period="YS",
    minor_amplitude=5,
    minor_seasonal_period="D"
)
n_jitter = len(df_jitter)

# nudge just 3 timestamps by a few minutes -- everything else stays exactly hourly
jitter_idx = np.random.choice(n_jitter, 3, replace=False)
df_jitter.loc[jitter_idx, "datetime"] += pd.to_timedelta(
    np.random.randint(1, 50, len(jitter_idx)), unit="m"
)

file_jitter = OUTPUT_DIR / "time_series_jitter_pitfall.csv"
df_jitter.to_csv(file_jitter, sep=";", index=False)

ts_jitter = TimeSeries(name="JitterPitfall", alias="pf")
ts_jitter.load_data(
    file_data=file_jitter,
    input_dtfield="datetime",
    input_varfield="level",
    in_sep=";",
)
print(f"Only {len(jitter_idx)} of {n_jitter} timestamps were nudged, "
      f"detected frequency is: {ts_jitter.dtfreq!r} (correctly hourly, "
      f"despite the jitter)")

Only 3 of 49 timestamps were nudged, detected frequency is: 'h' (correctly hourly, despite the jitter)

Frequency detection uses the mode of consecutive timestamp deltas rather than checking whether any minute value repeats, so a small minority of jittered timestamps doesn’t change the outcome – most deltas are still exactly 1 hour, and that’s what wins. Standardizing now produces the expected row count with no artifacts:

print(f"Rows before standardize: {len(ts_jitter.data)}")
ts_jitter.standardize()
print(f"Rows after standardize:  {len(ts_jitter.data)}")
ts_jitter.data.head(9)

Rows before standardize: 49
Rows after standardize:  73

	datetime	v
0	2020-01-01 00:00:00	102.592102
1	2020-01-01 01:00:00	103.513343
2	2020-01-01 02:00:00	99.538197
3	2020-01-01 03:00:00	102.020622
4	2020-01-01 04:00:00	104.179624
5	2020-01-01 05:00:00	105.861248
6	2020-01-01 06:00:00	103.070165
7	2020-01-01 07:00:00	109.329654
8	2020-01-01 08:00:00	106.803752

As a second, independent check: bucketing itself is now based on flooring each timestamp to its own slot (.dt.floor(offset)) rather than a hardcoded string pattern, so even if .dtfreq were forced to an incorrect value, each row would still land in its own correct slot instead of overwriting its neighbors. Forcing .dtfreq to "20min" directly demonstrates this:

import copy

ts_forced = copy.deepcopy(ts_jitter)
ts_forced.dtfreq = "20min"  # deliberately force the wrong frequency
ts_forced.standardize()

vf = ts_forced.varfield
hourly_groups = ts_forced.data.groupby(ts_forced.data[ts_forced.dtfield].dt.floor("h"))[vf]
distinct_per_hour = hourly_groups.nunique(dropna=False)
print("Distinct values per hour under a forced 20min grid "
      "(expect >1 -- slots bucket independently, no collapse):")
print(distinct_per_hour.value_counts())

Distinct values per hour under a forced 20min grid (expect >1 -- slots bucket independently, no collapse):
v
2    49
1    48
Name: count, dtype: int64

Most hours now show 2 distinct values (real data in two of the three 20-minute slots, NaN in the third), confirming the slots are independent rather than silently copied – the collapse artifact is gone.

Recap#

.standardize() forces the series onto a regular grid at .dtfreq, inserting NaN rows for any missing timestamp.
.is_standard tracks whether this has been done; many other methods (.get_epochs(), .interpolate_gaps(), .scale_up()) call .standardize() internally if needed.
Irregular data (missing rows) doesn’t show up as nulls until standardized – check timestamp deltas or row count against the expected range to catch it beforehand.
Frequency detection (mode of consecutive deltas) and epoch bucketing (.dt.floor()) are both robust to a handful of jittered timestamps – a small minority of off-grid rows won’t flip the whole series onto the wrong frequency or silently collapse values within a slot.