{ "cells": [ { "cell_type": "markdown", "id": "d1e2f30001", "metadata": {}, "source": "# Time Series - Downscaling" }, { "cell_type": "markdown", "id": "d1e2f30002", "metadata": {}, "source": [ "Downscaling is the opposite of upscaling: going from a coarser time step to a finer one. `.scale_down()` handles this differently depending on the series' `agg` attribute:\n", "\n", "- For **non-sum** variables (`agg` is `\"mean\"`, `\"max\"`, `\"min\"`, ...), it's plain linear interpolation between the original points.\n", "- For **sum** variables (`agg == \"sum\"`, e.g. daily precipitation totals), each source period's total must be *conserved*: the finer sub-steps within a day have to add back up to exactly that day's original total. Naive linear interpolation doesn't guarantee this -- it can blend values across day boundaries.\n", "\n", "This tutorial covers the `\"sum\"` case specifically, since it's the more interesting one:\n", "\n", "- **Uniform split** (the default): each day's total spread evenly across its sub-daily steps.\n", "- **Covariate-weighted**: each day's total shaped by a higher-resolution proxy signal (e.g. hourly satellite precipitation estimates), so the sub-daily *pattern* comes from the covariate while the sub-daily *total* still matches the original daily gauge value exactly.\n", "\n", "Both are visualized with markers so the sub-daily steps are clearly distinguishable from the original daily values.\n", "\n", "Like `.scale_up()`, `.scale_down()` takes an `inplace` parameter (default `False`): with `inplace=False` it returns a **new `TimeSeries` object** -- an exact copy of the parent holding the downscaled data -- rather than a bare DataFrame; with `inplace=True` it overwrites the current object's data and returns `None`." ] }, { "cell_type": "markdown", "id": "d1e2f30003", "metadata": {}, "source": [ "## Notebook setup\n", "\n", "For users running this tutorial as a Jupyter Notebook, this cell must be executed first:" ] }, { "cell_type": "code", "id": "d1e2f30004", "metadata": {}, "source": [ "import sys\n", "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "# Install `plans` in `google.colab`.\n", "# Use `pip install plans` for other environments.\n", "\n", "if \"google.colab\" in sys.modules:\n", " import os\n", " os.system(f\"{sys.executable} -m pip install -q plans\")\n", "\n", "# This avoids warnings related to uninstalled fonts\n", "import logging\n", "logging.getLogger('matplotlib.font_manager').setLevel(logging.ERROR)\n", "\n", "# Ensure figures render inline\n", "%matplotlib inline\n", "\n", "# define output folder\n", "OUTPUT_DIR = Path(\"outputs/time-series\")\n", "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n", "print(f\"Outputs will be saved to: ./{OUTPUT_DIR}\")\n", "\n", "RNG_SEED = 2\n", "np.random.seed(RNG_SEED)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30006", "metadata": {}, "source": [ "## Create a small daily precipitation series\n", "\n", "A tiny, deliberately small dataset (5 days) so every individual downscaled point is easy to see on a plot." ] }, { "metadata": {}, "cell_type": "code", "source": [ "from plans.datasets import TimeSeries\n", "\n", "dates = pd.date_range(\"2020-01-01\", periods=5, freq=\"D\")\n", "daily_totals = [3.6, 4.8, 0.8, 2.4, 1.9]\n", "\n", "df_daily = pd.DataFrame({\"datetime\": dates, \"precip\": daily_totals})\n", "file_daily = OUTPUT_DIR / \"precip_daily.csv\"\n", "df_daily.to_csv(file_daily, sep=\";\", index=False)\n", "\n", "ts_daily = TimeSeries(name=\"Daily gauge\", alias=\"gauge\")\n", "ts_daily.load_data(\n", " file_data=file_daily,\n", " input_dtfield=\"datetime\",\n", " input_varfield=\"precip\",\n", " in_sep=\";\",\n", ")\n", "# precipitation is a flow/sum variable -- totals must be conserved on downscale\n", "ts_daily.agg = \"sum\"\n", "\n", "print(f\"Daily totals: {daily_totals}\")\n", "print(f\"Sum of all days: {sum(daily_totals)}\")\n", "ts_daily.view()" ], "id": "47f53daae2cc9934", "outputs": [], "execution_count": null }, { "metadata": {}, "cell_type": "markdown", "source": [ "## Downscale with uniform split (default)\n", "\n", "With no `covariate` argument, `.scale_down()` splits each day's total evenly across its sub-daily steps." ], "id": "155f017af27a0f06" }, { "metadata": {}, "cell_type": "code", "source": [ "v = ts_daily.varfield\n", "ts_uniform = ts_daily.scale_down(freq=\"3h\")\n", "print(type(ts_uniform))\n", "ts_uniform.data.head(10)" ], "id": "ef26ff27734798f", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f3000a", "metadata": {}, "source": "Confirm each day's downscaled sub-steps still add up to that day's original total:" }, { "cell_type": "code", "id": "d1e2f3000b", "metadata": {}, "source": [ "df_uniform = ts_uniform.data.copy()\n", "df_uniform[\"day\"] = df_uniform[ts_uniform.dtfield].dt.date\n", "check = df_uniform.groupby(\"day\")[v].sum().reset_index()\n", "check[\"original\"] = daily_totals\n", "check" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f3000c", "metadata": {}, "source": "Visualizing: original daily totals as large markers, downscaled 3-hourly sub-steps as small markers on a connecting line. **The daily-total marker is placed at the end of the day it represents**, not the start -- a day's rainfall total is only fully known once the day is over, so plotting it at midnight (the day's start timestamp used internally) would visually suggest the rain fell *before* it was recorded. Note the black squares sit *above* the blue line -- they represent each day's **total**, while the blue points are each day's total divided into 8 equal 3-hourly shares, so individually they're much smaller than the day's total. Each day's flat segment visually confirms the uniform split, and vertical guide lines mark the day boundaries:" }, { "metadata": {}, "cell_type": "code", "source": [ "fig, ax = plt.subplots(figsize=(6, 3))\n", "\n", "ax.plot(\n", " df_uniform[\"datetime\"], df_uniform[v],\n", " marker=\"o\", markersize=4, linewidth=1, color=\"tab:blue\",\n", " label=\"3h downscaled (uniform)\"\n", ")\n", "# plot the daily-total marker at the END of the day it represents,\n", "# not its internal start-of-day timestamp -- purely a plotting choice,\n", "# the underlying ts_daily.data timestamps are unchanged\n", "one_day = pd.Timedelta(days=1)\n", "ax.plot(\n", " ts_daily.data[ts_daily.dtfield] + one_day, ts_daily.data[v],\n", " marker=\"s\", markersize=6, linestyle=\"None\", color=\"black\",\n", " label=\"Original daily total (plotted at day's end)\"\n", ")\n", "for d in ts_daily.data[ts_daily.dtfield]:\n", " ax.axvline(d, color=\"gray\", linestyle=\":\", linewidth=0.8, alpha=0.6)\n", "\n", "ax.set_title(\"Uniform-split downscale: daily -> 3-hourly\")\n", "ax.set_xlabel(\"datetime\")\n", "ax.set_ylabel(\"precip\")\n", "ax.legend()\n", "plt.tight_layout()\n", "plt.show()" ], "id": "c6673bfb8a3fd78c", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f3000e", "metadata": {}, "source": [ "Note the discontinuity at each day boundary -- the uniform split has no information about *when* within a day the rain actually fell, so it produces a flat, physically implausible step function. This is where a covariate helps." ] }, { "cell_type": "markdown", "id": "d1e2f3000f", "metadata": {}, "source": [ "## Downscale with a covariate\n", "\n", "Now suppose we have a higher-resolution proxy signal -- e.g. hourly satellite-derived precipitation intensity. It's *not* a direct measurement (its own units and totals don't need to match the gauge), but its **shape** over time tells us when, within each day, precipitation was more or less intense. For this example, each day has a single, brief rain pulse at a different hour -- a simple, clearly visible case for seeing how the shape carries through to the downscaled output.\n", "\n", "`.scale_down(freq=..., covariate=ts_satellite)` uses exactly that shape: within each day, sub-steps get a share of the day's total proportional to the covariate's value at that sub-step, so the sub-daily *pattern* comes from the covariate while the sub-daily *total* still matches the gauge exactly." ] }, { "metadata": {}, "cell_type": "code", "source": [ "# synthetic hourly satellite-like signal: a single narrow rain pulse\n", "# per day, at a randomized (but reproducible) hour, near-zero elsewhere\n", "hourly_idx = pd.date_range(\"2020-01-01\", \"2020-01-06\", freq=\"1h\", inclusive=\"left\")\n", "n_days = len(daily_totals)\n", "pulse_hours = np.random.uniform(6, 20, n_days) # each day's pulse falls between 06:00-20:00\n", "pulse_width = 1.0 # hours -- narrower means a sharper, more concentrated pulse\n", "\n", "hours_of_day = (hourly_idx.hour + hourly_idx.minute / 60).values\n", "day_of = np.array([(t.normalize() - hourly_idx[0].normalize()).days for t in hourly_idx])\n", "\n", "signal = np.zeros(len(hourly_idx))\n", "for d in range(n_days):\n", " mask = day_of == d\n", " signal[mask] = np.exp(-0.5 * ((hours_of_day[mask] - pulse_hours[d]) / pulse_width) ** 2)\n", "\n", "df_satellite = pd.DataFrame({\"datetime\": hourly_idx, \"intensity\": signal})\n", "file_satellite = OUTPUT_DIR / \"precip_satellite.csv\"\n", "df_satellite.to_csv(file_satellite, sep=\";\", index=False)\n", "\n", "ts_satellite = TimeSeries(name=\"Satellite proxy\", alias=\"sat\")\n", "ts_satellite.load_data(\n", " file_data=file_satellite,\n", " input_dtfield=\"datetime\",\n", " input_varfield=\"intensity\",\n", " in_sep=\";\",\n", ")\n", "print(f\"Covariate rows: {len(ts_satellite.data)}, detected frequency: {ts_satellite.dtfreq}\")\n", "print(f\"Pulse hour per day: {np.round(pulse_hours, 1).tolist()}\")\n", "\n", "ts_satellite.view()" ], "id": "c4a3e6258d02021b", "outputs": [], "execution_count": null }, { "metadata": {}, "cell_type": "markdown", "source": "The gauge target is `3h`, finer than the covariate's own `1h`; `.scale_down()` linearly interpolates the covariate onto the target grid internally before using it as weights -- no separate resampling step is needed on the user's side.", "id": "be10ae6f140c165c" }, { "metadata": {}, "cell_type": "code", "source": [ "ts_covariate = ts_daily.scale_down(freq=\"3h\", covariate=ts_satellite)\n", "print(type(ts_covariate))\n", "ts_covariate.data.head(10)" ], "id": "d22397afa76bdaae", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30013", "metadata": {}, "source": "Same conservation check as before -- covariate-weighting still preserves each day's original total exactly:" }, { "cell_type": "code", "id": "d1e2f30014", "metadata": {}, "source": [ "df_covariate = ts_covariate.data.copy()\n", "df_covariate[\"day\"] = df_covariate[ts_covariate.dtfield].dt.date\n", "check_cov = df_covariate.groupby(\"day\")[v].sum().reset_index()\n", "check_cov[\"original\"] = daily_totals\n", "check_cov" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30014a", "metadata": {}, "source": [ "## A note on the return type: `inplace`\n", "\n", "Every call above used the default `inplace=False`, which is why `ts_uniform` and `ts_covariate` are full `TimeSeries` objects (not DataFrames) -- notice `.data` was needed to get at the underlying table. Setting `inplace=True` instead overwrites the object's own data and returns `None`:" ] }, { "cell_type": "code", "id": "d1e2f30014b", "metadata": {}, "source": [ "ts_demo = TimeSeries(name=\"Demo\", alias=\"demo\")\n", "ts_demo.load_data(\n", " file_data=file_daily,\n", " input_dtfield=\"datetime\",\n", " input_varfield=\"precip\",\n", " in_sep=\";\",\n", ")\n", "ts_demo.agg = \"sum\"\n", "\n", "print(f\"Rows before: {len(ts_demo.data)}\")\n", "return_value = ts_demo.scale_down(freq=\"3h\", inplace=True)\n", "print(f\"Return value: {return_value!r}\")\n", "print(f\"Rows after (same object, overwritten): {len(ts_demo.data)}\")" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30015", "metadata": {}, "source": "Visualizing all three together: original daily totals, the uniform-split downscale, and the covariate-weighted downscale -- note how the covariate version now has a plausible, physically-informed within-day shape instead of flat steps, while still returning to the same daily totals as the uniform version:" }, { "metadata": {}, "cell_type": "code", "source": [ "fig, ax = plt.subplots(figsize=(6, 3))\n", "\n", "ax.plot(\n", " df_uniform[\"datetime\"], df_uniform[v],\n", " marker=\"o\", markersize=3, linewidth=1, color=\"tab:blue\", alpha=0.6,\n", " label=\"Uniform split\"\n", ")\n", "ax.plot(\n", " df_covariate[\"datetime\"], df_covariate[v],\n", " marker=\"^\", markersize=4, linewidth=1.2, color=\"tab:green\",\n", " label=\"Covariate-weighted\"\n", ")\n", "one_day = pd.Timedelta(days=1)\n", "ax.plot(\n", " ts_daily.data[ts_daily.dtfield] + one_day, ts_daily.data[v],\n", " marker=\"s\", markersize=6, linestyle=\"None\", color=\"black\",\n", " label=\"Original daily total (plotted at day's end)\"\n", ")\n", "for d in ts_daily.data[ts_daily.dtfield]:\n", " ax.axvline(d, color=\"gray\", linestyle=\":\", linewidth=0.8, alpha=0.6)\n", "\n", "ax.set_title(\"Downscaling daily precipitation: uniform vs. covariate-weighted\")\n", "ax.set_xlabel(\"datetime\")\n", "ax.set_ylabel(\"precip\")\n", "ax.legend()\n", "plt.tight_layout()\n", "plt.show()" ], "id": "36119f3b5dd8ef72", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30017", "metadata": {}, "source": [ "## Downscaling a non-sum variable: water level\n", "\n", "Precipitation is a flow variable -- its totals must be conserved. Something like a daily-average water level is different: it's not additive, so there's no \"total\" to preserve. For this case, `.scale_down()` takes a different approach:\n", "\n", "1. Each coarse data point is repositioned in time according to the `align` parameter -- `\"start\"`, `\"center\"` (default), or `\"end\"` -- reflecting where that value is assumed to sit within the period it represents. A daily average, for instance, is more representative of *midday* than midnight, so `\"center\"` anchors it at `12:00`.\n", "2. The repositioned points are linearly interpolated onto the fine grid.\n", "3. A single global multiplicative correction is applied so the **downscaled series' overall mean** matches the **original series' overall mean** -- not each individual day's mean, just the series as a whole. (An earlier design that tried to preserve *each day's* mean exactly was tested and rejected -- it produced wild oscillations with volatile data. See the aside below.)" ] }, { "metadata": {}, "cell_type": "code", "source": [ "dates_level = pd.date_range(\"2020-01-01\", periods=10, freq=\"D\")\n", "daily_means = [15.0, 20.0, 15.0, 12.0, 11.0, 9.0, 8.5, 9.5, 7.2, 6.4]\n", "\n", "df_level = pd.DataFrame({\"datetime\": dates_level, \"level\": daily_means})\n", "file_level = OUTPUT_DIR / \"level_daily.csv\"\n", "df_level.to_csv(file_level, sep=\";\", index=False)\n", "\n", "ts_level = TimeSeries(name=\"Reservoir level\", alias=\"lvl\")\n", "ts_level.load_data(\n", " file_data=file_level,\n", " input_dtfield=\"datetime\",\n", " input_varfield=\"level\",\n", " in_sep=\";\",\n", ")\n", "print(f\"agg (default): {ts_level.agg!r}\") # \"mean\" -- not \"sum\", so align/correction path applies\n", "\n", "vl = ts_level.varfield\n", "ts_center = ts_level.scale_down(freq=\"3h\") # align=\"center\" is the default\n", "ts_start = ts_level.scale_down(freq=\"3h\", align=\"start\")\n", "ts_end = ts_level.scale_down(freq=\"3h\", align=\"end\")\n", "\n", "for name, out in ((\"center (default)\", ts_center), (\"start\", ts_start), (\"end\", ts_end)):\n", " print(f\"align={name:<17} overall mean: {out.data[vl].mean():.6f} \"\n", " f\"(original: {sum(daily_means) / len(daily_means):.6f})\")" ], "id": "2bbc0d057329d494", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30019", "metadata": {}, "source": "Visualizing all three `align` options together, with the original daily-mean markers placed at the timestamp each `align` option anchors them to -- `\"start\"` at midnight, `\"center\"` at midday, `\"end\"` at the next midnight. Notice how each curve passes closer to its own markers, since those are exactly the points it interpolates through:" }, { "metadata": {}, "cell_type": "code", "source": [ "df_start = ts_start.data\n", "df_center = ts_center.data\n", "df_end = ts_end.data\n", "\n", "fig, ax = plt.subplots(figsize=(6, 3))\n", "\n", "ax.plot(df_start[\"datetime\"], df_start[vl], marker=\"o\", markersize=3,\n", " linewidth=1, color=\"tab:blue\", alpha=0.7, label=\"align='start'\")\n", "ax.plot(df_center[\"datetime\"], df_center[vl], marker=\"^\", markersize=3,\n", " linewidth=1.2, color=\"tab:purple\", label=\"align='center' (default)\")\n", "ax.plot(df_end[\"datetime\"], df_end[vl], marker=\"v\", markersize=3,\n", " linewidth=1, color=\"tab:red\", alpha=0.7, label=\"align='end'\")\n", "\n", "half_day = pd.Timedelta(hours=12)\n", "one_day = pd.Timedelta(days=1)\n", "ax.plot(ts_level.data[ts_level.dtfield], ts_level.data[vl],\n", " marker=\"s\", markersize=6, linestyle=\"None\", color=\"tab:blue\", label=\"daily mean (start anchor)\")\n", "ax.plot(ts_level.data[ts_level.dtfield] + half_day, ts_level.data[vl],\n", " marker=\"D\", markersize=6, linestyle=\"None\", color=\"tab:purple\", label=\"daily mean (center anchor)\")\n", "ax.plot(ts_level.data[ts_level.dtfield] + one_day, ts_level.data[vl],\n", " marker=\"P\", markersize=6, linestyle=\"None\", color=\"tab:red\", label=\"daily mean (end anchor)\")\n", "\n", "for d in ts_level.data[ts_level.dtfield]:\n", " ax.axvline(d, color=\"gray\", linestyle=\":\", linewidth=0.8, alpha=0.5)\n", "\n", "ax.set_title(\"Downscaling a non-sum (mean) variable: effect of `align`\")\n", "ax.set_xlabel(\"datetime\")\n", "ax.set_ylabel(\"level\")\n", "ax.set_ylim(0, 25)\n", "ax.legend(fontsize=8)\n", "\n", "plt.tight_layout()\n", "plt.show()" ], "id": "72740791f1bdf508", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f3001b", "metadata": {}, "source": [ "### Aside: why not preserve each day's mean exactly?\n", "\n", "It's tempting to want a stricter guarantee -- that each individual day's sub-daily average, not just the whole series', matches its reported daily mean. There's a simple technique for this: place unknown boundary values $S_i$ between periods such that $(S_i + S_{i+1})/2$ equals each period's mean exactly, then linearly interpolate between the $S_i$. This *can* be solved exactly (it's a linear recurrence), but it's dangerously unstable -- a volatile sequence of daily means can force boundary values far outside the plausible range of the variable:" ] }, { "cell_type": "code", "id": "d1e2f3001c", "metadata": {}, "source": [ "volatile_means = np.array([10.0, 40.0, 8.0, 35.0, 12.0])\n", "S = np.empty(len(volatile_means) + 1)\n", "S[0] = volatile_means[0]\n", "for i, m in enumerate(volatile_means):\n", " S[i + 1] = 2 * m - S[i]\n", "print(f\"Daily means: {volatile_means.tolist()}\")\n", "print(f\"Boundary values: {np.round(S, 1).tolist()}\")\n", "print(\"Note the boundary values swing far outside the range of the original \"\n", " \"data (even negative) -- this is why `.scale_down()` uses the simpler, \"\n", " \"more stable whole-series mean correction instead.\")" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f3001d", "metadata": {}, "source": [ "### Any `agg` other than `\"sum\"` follows the same path\n", "\n", "The align/correction behavior isn't specific to `\"mean\"` -- it applies to any `agg` value that isn't `\"sum\"`, since the branch check is simply `self.agg != \"sum\"`. Setting `agg` to something like `\"max\"` still runs without error, though it's worth noting the correction always targets the **mean** regardless of what `agg` conceptually represents -- there's no attempt to conserve a maximum, since a maximum isn't additive and can't be redistributed the way a sum can:" ] }, { "cell_type": "code", "id": "d1e2f3001e", "metadata": {}, "source": [ "ts_level.agg = \"max\" # arbitrary non-sum agg\n", "ts_max_down = ts_level.scale_down(freq=\"6h\")\n", "print(type(ts_max_down))\n", "ts_max_down.data.head(6)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "d1e2f30019a", "metadata": {}, "source": [ "## Recap\n", "\n", "- `.scale_down(freq)` returns a new `TimeSeries` object by default (`inplace=False`); pass `inplace=True` to overwrite the current object's data instead and get `None` back.\n", "- For a `\"sum\"`-aggregated series, `.scale_down(freq)` uses **uniform split** by default: each source period's total is spread evenly across its finer sub-steps, exactly preserving that period's total.\n", "- `.scale_down(freq, covariate=ts_other)` shapes that distribution using a higher-resolution proxy series instead of splitting evenly -- the covariate is auto-interpolated onto the target frequency if needed, and falls back to uniform weighting in any period where the covariate is entirely zero or missing.\n", "- Both `\"sum\"` modes exactly conserve each source period's original total -- verified directly by grouping the downscaled output back to the source resolution.\n", "- For any `agg` other than `\"sum\"` (`\"mean\"`, `\"max\"`, `\"min\"`, ...), `.scale_down(freq, align=...)` repositions each coarse point per `align` (`\"start\"`, `\"center\"` default, or `\"end\"`), linearly interpolates, then applies a single global correction so the downscaled series' **overall** mean matches the original -- not each individual period's mean, which turned out to be an unstable guarantee to chase exactly." ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }