{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50001",
   "metadata": {},
   "source": "# Time Series - Standardizing"
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50002",
   "metadata": {},
   "source": [
    "Real time series data doesn't always land on a clean, evenly-spaced grid: a sensor logger can skip a reading, drift by a few minutes, or simply have missing rows rather than explicit nulls. The `.standardize()` method forces the series onto a regular grid at its detected frequency -- inserting **null rows** wherever a timestamp is missing from that grid.\n",
    "\n",
    "This is a foundational step: many other methods (`.get_epochs()`, `.interpolate_gaps()`, `.scale_up()`) assume a regular grid and call `.standardize()` internally if the series isn't already standard. Understanding what `.standardize()` does -- and where it can surprise you -- makes those methods much easier to reason about.\n",
    "\n",
    "Covered here:\n",
    "\n",
    "- What \"irregular\" data looks like (missing rows on an otherwise regular grid)\n",
    "- Frequency detection (`.dtfreq`) and the `.is_standard` flag\n",
    "- What `.standardize()` actually does to row count and null count\n",
    "- A caveat: how minute-level jitter can silently change the detected frequency and distort the result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50003",
   "metadata": {},
   "source": [
    "## Notebook setup\n",
    "\n",
    "For users running this tutorial as a Jupyter Notebook, this cell must be executed first:"
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50004",
   "metadata": {},
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "import pprint\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Install `plans` in `google.colab`.\n",
    "# Use `pip install plans` for other environments.\n",
    "\n",
    "if \"google.colab\" in sys.modules:\n",
    "    import os\n",
    "    os.system(f\"{sys.executable} -m pip install -q plans\")\n",
    "\n",
    "# This avoids warnings related to uninstalled fonts\n",
    "import logging\n",
    "logging.getLogger('matplotlib.font_manager').setLevel(logging.ERROR)\n",
    "\n",
    "# define output folder\n",
    "OUTPUT_DIR = Path(\"outputs/time-series\")\n",
    "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
    "print(f\"Outputs will be saved to: ./{OUTPUT_DIR}\")\n",
    "\n",
    "# fixed seed so the irregular timestamps are reproducible\n",
    "RNG_SEED = 11\n",
    "np.random.seed(RNG_SEED)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50006",
   "metadata": {},
   "source": [
    "## Create a perfect synthetic series\n",
    "\n",
    "Same starting point as the other tutorials: a Trend-Seasonality-Noise archetype series, hourly resolution, two weeks of data."
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50007",
   "metadata": {},
   "source": [
    "from plans.datasets import TimeSeries\n",
    "\n",
    "df_perfect = TimeSeries.make_synthetic_tsn(\n",
    "    start=\"2020-01-01\",\n",
    "    end=\"2020-01-15\",\n",
    "    base=100,\n",
    "    freq=\"1h\",\n",
    "    trend=0.001,\n",
    "    noise_sd=4.0,\n",
    "    amplitude=50,\n",
    "    seasonal_period=\"YS\",\n",
    "    minor_amplitude=20,\n",
    "    minor_seasonal_period=\"D\"\n",
    ")\n",
    "print(f\"Rows: {len(df_perfect)}\")\n",
    "df_perfect.head()"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50008",
   "metadata": {},
   "source": [
    "## Make the grid irregular\n",
    "\n",
    "This is different from the gaps tutorial: there, values were set to `NaN` but every timestamp still existed as a row. Here, we **drop rows entirely** -- the timestamps themselves go missing, so the grid is no longer evenly spaced. This mimics a logger that occasionally fails to write a record at all, rather than writing a bad value."
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50009",
   "metadata": {},
   "source": [
    "n = len(df_perfect)\n",
    "\n",
    "# -- drop ~8% of rows entirely (scattered missing timestamps) --\n",
    "DROP_FRACTION = 0.08\n",
    "drop_mask = np.random.rand(n) < DROP_FRACTION\n",
    "df_irregular = df_perfect[~drop_mask].reset_index(drop=True)\n",
    "\n",
    "print(f\"Original rows: {n}\")\n",
    "print(f\"Rows after dropping: {len(df_irregular)} \"\n",
    "      f\"({n - len(df_irregular)} timestamps missing entirely)\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f5000a",
   "metadata": {},
   "source": "A quick check on the timestamp deltas confirms the grid is no longer uniform -- most gaps are 1 hour, but some are 2 hours or more where a row was dropped:"
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f5000b",
   "metadata": {},
   "source": [
    "deltas = df_irregular[\"datetime\"].diff().dropna()\n",
    "print(deltas.value_counts().sort_index())"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f5000c",
   "metadata": {},
   "source": [
    "## Save and load through the `TimeSeries` object\n",
    "\n",
    "Same workflow as the other tutorials: export to CSV, then load through `.load_data()`."
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f5000d",
   "metadata": {},
   "source": [
    "file_csv = OUTPUT_DIR / \"time_series_irregular.csv\"\n",
    "df_irregular.to_csv(file_csv, sep=\";\", index=False)\n",
    "print(f\"Saved to: {file_csv}\")\n",
    "\n",
    "ts = TimeSeries(name=\"Irregular\", alias=\"irr\")\n",
    "ts.load_data(\n",
    "    file_data=file_csv,\n",
    "    input_dtfield=\"datetime\",\n",
    "    input_varfield=\"level\",\n",
    "    in_sep=\";\",\n",
    ")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f5000e",
   "metadata": {},
   "source": [
    "## Inspect before standardizing\n",
    "\n",
    "Two attributes matter here:\n",
    "\n",
    "- **`.dtfreq`** -- the frequency `plans` detected from the timestamps (e.g. `\"h\"` for hourly). This is inferred from how much variation exists in the seconds/minutes/hours/day components of the data -- see the caveat section below for how this can go wrong.\n",
    "- **`.is_standard`** -- `False` until `.standardize()` has been called; tells you whether the series currently sits on a regular grid."
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f5000f",
   "metadata": {},
   "source": [
    "v = ts.varfield\n",
    "print(f\"Rows loaded: {len(ts.data)}\")\n",
    "print(f\"Detected frequency (.dtfreq): {ts.dtfreq}\")\n",
    "print(f\"Is standard (.is_standard): {ts.is_standard}\")\n",
    "print(f\"Null records: {ts.data[v].isna().sum()}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50010",
   "metadata": {},
   "source": [
    "Note that null count is zero -- the irregularity here is entirely about *missing rows*, not missing values. `.view()` on ungapped-but-irregular data won't visually show anything obviously wrong; the problem only becomes apparent once you look at the timestamp spacing (as above) or once resampling/aggregation methods run into the uneven grid."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50011",
   "metadata": {},
   "source": [
    "## Standardize\n",
    "\n",
    "`.standardize()` builds a full, regular date range at `.dtfreq` spanning the series' start to end, and left-merges the existing data onto it. Any regular-grid slot with no matching data becomes a null row. It updates `self.data` in place and sets `.is_standard = True`."
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50012",
   "metadata": {},
   "source": [
    "rows_before = len(ts.data)\n",
    "ts.standardize()\n",
    "rows_after = len(ts.data)\n",
    "\n",
    "print(f\"Rows before standardize: {rows_before}\")\n",
    "print(f\"Rows after standardize:  {rows_after}\")\n",
    "print(f\"Null records introduced: {ts.data[v].isna().sum()}\")\n",
    "print(f\"Is standard now: {ts.is_standard}\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50013",
   "metadata": {},
   "source": "The row count now matches a full regular hourly range, and every missing timestamp from the original irregular data has become an explicit `NaN` row -- exactly the kind of gap the earlier gaps/epochs tutorial works with. Confirming the grid is now evenly spaced:"
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50014",
   "metadata": {},
   "source": [
    "deltas_std = ts.data[ts.dtfield].diff().dropna()\n",
    "print(deltas_std.value_counts())"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50015",
   "metadata": {},
   "source": [
    "## Robustness check: minute-level jitter\n",
    "\n",
    "Frequency detection (`.dtfreq`) is inferred from timestamp spacing, and `standardize()`'s bucketing floors each row independently onto that frequency's grid. Both of these matter for irregular data: a naive detector based on \"do the minutes vary at all\" would be thrown off by a single jittered timestamp, and naive bucketing based on hardcoded string patterns could accidentally merge rows that should stay in separate slots.\n",
    "\n",
    "The next cell stress-tests this on a small, isolated example -- three timestamps out of 49 nudged by a random number of minutes, everything else exactly hourly:"
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50016",
   "metadata": {},
   "source": [
    "df_jitter = TimeSeries.make_synthetic_tsn(\n",
    "    start=\"2020-01-01\",\n",
    "    end=\"2020-01-03\",\n",
    "    base=100,\n",
    "    freq=\"1h\",\n",
    "    trend=0.0,\n",
    "    noise_sd=2.0,\n",
    "    amplitude=10,\n",
    "    seasonal_period=\"YS\",\n",
    "    minor_amplitude=5,\n",
    "    minor_seasonal_period=\"D\"\n",
    ")\n",
    "n_jitter = len(df_jitter)\n",
    "\n",
    "# nudge just 3 timestamps by a few minutes -- everything else stays exactly hourly\n",
    "jitter_idx = np.random.choice(n_jitter, 3, replace=False)\n",
    "df_jitter.loc[jitter_idx, \"datetime\"] += pd.to_timedelta(\n",
    "    np.random.randint(1, 50, len(jitter_idx)), unit=\"m\"\n",
    ")\n",
    "\n",
    "file_jitter = OUTPUT_DIR / \"time_series_jitter_pitfall.csv\"\n",
    "df_jitter.to_csv(file_jitter, sep=\";\", index=False)\n",
    "\n",
    "ts_jitter = TimeSeries(name=\"JitterPitfall\", alias=\"pf\")\n",
    "ts_jitter.load_data(\n",
    "    file_data=file_jitter,\n",
    "    input_dtfield=\"datetime\",\n",
    "    input_varfield=\"level\",\n",
    "    in_sep=\";\",\n",
    ")\n",
    "print(f\"Only {len(jitter_idx)} of {n_jitter} timestamps were nudged, \"\n",
    "      f\"detected frequency is: {ts_jitter.dtfreq!r} (correctly hourly, \"\n",
    "      f\"despite the jitter)\")"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50017",
   "metadata": {},
   "source": "Frequency detection uses the *mode* of consecutive timestamp deltas rather than checking whether any minute value repeats, so a small minority of jittered timestamps doesn't change the outcome -- most deltas are still exactly 1 hour, and that's what wins. Standardizing now produces the expected row count with no artifacts:"
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f50018",
   "metadata": {},
   "source": [
    "print(f\"Rows before standardize: {len(ts_jitter.data)}\")\n",
    "ts_jitter.standardize()\n",
    "print(f\"Rows after standardize:  {len(ts_jitter.data)}\")\n",
    "ts_jitter.data.head(9)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f50019",
   "metadata": {},
   "source": [
    "As a second, independent check: bucketing itself is now based on flooring each timestamp to its own slot (`.dt.floor(offset)`) rather than a hardcoded string pattern, so even if `.dtfreq` were forced to an incorrect value, each row would still land in its own correct slot instead of overwriting its neighbors. Forcing `.dtfreq` to `\"20min\"` directly demonstrates this:"
   ]
  },
  {
   "cell_type": "code",
   "id": "b1c2d3e4f5001a",
   "metadata": {},
   "source": [
    "import copy\n",
    "\n",
    "ts_forced = copy.deepcopy(ts_jitter)\n",
    "ts_forced.dtfreq = \"20min\"  # deliberately force the wrong frequency\n",
    "ts_forced.standardize()\n",
    "\n",
    "vf = ts_forced.varfield\n",
    "hourly_groups = ts_forced.data.groupby(ts_forced.data[ts_forced.dtfield].dt.floor(\"h\"))[vf]\n",
    "distinct_per_hour = hourly_groups.nunique(dropna=False)\n",
    "print(\"Distinct values per hour under a forced 20min grid \"\n",
    "      \"(expect >1 -- slots bucket independently, no collapse):\")\n",
    "print(distinct_per_hour.value_counts())"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f5001b_note",
   "metadata": {},
   "source": [
    "Most hours now show 2 distinct values (real data in two of the three 20-minute slots, `NaN` in the third), confirming the slots are independent rather than silently copied -- the collapse artifact is gone."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1c2d3e4f5001b",
   "metadata": {},
   "source": [
    "## Recap\n",
    "\n",
    "- `.standardize()` forces the series onto a regular grid at `.dtfreq`, inserting `NaN` rows for any missing timestamp.\n",
    "- `.is_standard` tracks whether this has been done; many other methods (`.get_epochs()`, `.interpolate_gaps()`, `.scale_up()`) call `.standardize()` internally if needed.\n",
    "- Irregular data (missing rows) doesn't show up as nulls until standardized -- check timestamp deltas or row count against the expected range to catch it beforehand.\n",
    "- Frequency detection (mode of consecutive deltas) and epoch bucketing (`.dt.floor()`) are both robust to a handful of jittered timestamps -- a small minority of off-grid rows won't flip the whole series onto the wrong frequency or silently collapse values within a slot."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}