Time Series - Collections#

This tutorial builds a minimal TimeSeriesCollection for one (synthetic) catchment with five sensors that deliberately differ along every axis TimeSeriesCollection has to manage:

Series

Frequency

Range

Gaps

ppt (rainfall)

daily

full year

1 small (3d) + 1 large (20d)

Q1 (streamflow)

daily

Mar–Nov (shorter)

1 small (2d)

Q2 (streamflow)

daily

Mar–Nov (shorter)

1 small (2d)

tas (air temp.)

hourly

Jan–Jun (shorter, different unit of time)

1 small (5h) + 1 large (72h)

et (evapotransp.)

daily

full year

2 separate medium gaps (10d, 15d)

Covered here:

  • Building the five series with .generate(), from the default parameters of each child class

  • Loading them into one TimeSeriesCollection

  • standardize() and what it does to each series

  • Cross-series overlap via get_epochs()

  • Per-series gap/epoch structure via merge_local_epochs()

Notebook setup#

For users running this tutorial as a Jupyter Notebook, this cell must be executed first:

import sys
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Install `plans` in `google.colab`.
# Use `pip install plans` for other environments.

if "google.colab" in sys.modules:
    import os
    os.system(f"{sys.executable} -m pip install -q plans")

from plans.datasets.core import TimeSeries, TimeSeriesCollection

# This avoids warnings related to uninstalled fonts
import logging
logging.getLogger('matplotlib.font_manager').setLevel(logging.ERROR)

OUTPUT_DIR = Path("outputs/time-series-collection")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
print(f"Outputs will be saved to: ./{OUTPUT_DIR}")

RNG_SEED = 1
np.random.seed(RNG_SEED)

Outputs will be saved to: ./outputs/time-series-collection

Build synthetic series#

from plans.datasets import DischargeSeries, ETSeries, RainSeries, TemperatureSeries

df_p = RainSeries(name="Rain").generate(start="2019-01-01", end="2019-12-31", freq="D", seed=1)
df_p.loc[20:22, "ppt"] = np.nan     # small gap (3d)
df_p.loc[150:169, "ppt"] = np.nan   # large gap (20d) -> a real Epoch 0 gap

df_q = DischargeSeries(name="Flow").generate(start="2019-03-01", end="2019-11-30", freq="D", seed=2)
df_q.loc[40:41, "Q"] = np.nan     # small gap (2d)

df_q2 = DischargeSeries(name="Flow2").generate(start="2019-03-20", end="2019-11-30", freq="D", seed=3)
df_q2.loc[46:50, "Q"] = np.nan     # small gap (2d)

df_t = TemperatureSeries(name="AirTemp").generate(start="2019-01-01", end="2019-06-30", freq="h", seed=4)
df_t.loc[100:104, "tas"] = np.nan     # small gap (5h)
df_t.loc[2000:2071, "tas"] = np.nan   # large gap (72h)

df_et = ETSeries(name="ET").generate(start="2019-01-01", end="2019-12-31", freq="D", seed=5)
df_et.loc[60:69, "et"] = np.nan     # medium gap (10d)
df_et.loc[300:314, "et"] = np.nan   # medium gap (15d), separate from the first

for df, fname in [(df_p, "rain.csv"), (df_q, "flow.csv"), (df_q2, "flow2.csv"), (df_t, "temp.csv"), (df_et, "et.csv")]:
    df.to_csv(OUTPUT_DIR / fname, index=False, sep=";")

print("5 series written to", OUTPUT_DIR)

5 series written to outputs/time-series-collection

Load into a Time Series Collection#

TimeSeriesCollection reads one info table describing every series (name, alias, file, variable/datetime columns, units, coordinates):

info = pd.DataFrame({
    "Name": ["Rain", "Flow", "Flow2", "AirTemp", "ET"],
    "Alias": ["P1", "Q1", "Q2", "T1", "ET1"],
    "File": [str(OUTPUT_DIR / f) for f in ["rain.csv", "flow.csv", "flow2.csv", "temp.csv", "et.csv"]],
    "VarField": ["ppt", "Q", "Q", "tas", "et"],
    "DtField": ["datetime"] * 5,
    "Units": ["mm", "m3/s", "m3/s", "C", "mm"],
    "X": [0, 0, 0, 0, 0],
    "Y": [0, 0, 0, 0, 0],
    "Code": ["P001", "Q001", "Q002", "T001", "ET001"],
    "Source": ["synthetic"] * 5,
    "Description": [
        "daily rainfall",
        "daily streamflow", "daily streamflow",
        "hourly air temperature",
        "daily evapotranspiration"
    ],
    "Color": ["tab:blue", "tab:purple", "tab:purple", "tab:red", "tab:orange"],
})
info_file = OUTPUT_DIR / "info.csv"
info.to_csv(info_file, index=False, sep=";")

tsc = TimeSeriesCollection(name="MultiSensorCatchment")
tsc.load_data(table_file=str(info_file))
tsc.catalog
name alias size color source description units code x y ... datetime_res start end is_standard gap_size n_epoch n_gaps_small file_data_datetime_field file_data_variable_field file_data
0 AirTemp T1 4321 tab:red synthetic hourly air temperature C T001 0 0 ... hour 2019-01-01 2019-06-30 False 6 None None datetime tas /home/runner/work/plans/plans/docs/tutorials/o...
1 ET ET1 365 tab:orange synthetic daily evapotranspiration mm ET001 0 0 ... day 2019-01-01 2019-12-31 False 6 None None datetime et /home/runner/work/plans/plans/docs/tutorials/o...
2 Flow Q1 275 tab:purple synthetic daily streamflow m3/s Q001 0 0 ... day 2019-03-01 2019-11-30 False 6 None None datetime Q /home/runner/work/plans/plans/docs/tutorials/o...
3 Flow2 Q2 256 tab:purple synthetic daily streamflow m3/s Q002 0 0 ... day 2019-03-20 2019-11-30 False 6 None None datetime Q /home/runner/work/plans/plans/docs/tutorials/o...
4 Rain P1 365 tab:blue synthetic daily rainfall mm P001 0 0 ... day 2019-01-01 2019-12-31 False 6 None None datetime ppt /home/runner/work/plans/plans/docs/tutorials/o...

5 rows × 29 columns

Raw series at a glance#

Visualize individual TimeSeries accessing by the name

tsc.collection["Flow"].view()
tsc.collection["Flow2"].view()
../_images/a6fb2296279ffc4487a19a8dede7dac9a25ad51d02209a8e54990a6c0ecec462.png ../_images/e5df611a3e0079e19d155f2b0d32206e9d81240bbc0b75d7daa1db50faa6d5fc.png 
tsc.collection["Rain"].view()
tsc.collection["AirTemp"].view()
tsc.collection["ET"].view()
../_images/a04a2430d98d327e33812fa92c25e13612b106c69318fa0019ca2abdc61e1233.png ../_images/c0f81465fcda2747912ff5aa33768c08e55767579e263739577317b9e135df20.png ../_images/9fdd9192e3444ee4aeb5c2cda532c2106e901ae7d072fdfd6ff8b3e722d31035.png

Learn about the epochs#

Visualize epochs per Time Series

tsc.collection["ET"].view_specs["layout"] = "epochs"
tsc.collection["ET"].view()
../_images/68991724ab049459929d2c1e26a3aff29a5b3c321916fa4e0141a534c11ee4b7.png 
tsc.collection["Rain"].view_specs["layout"] = "epochs"
tsc.collection["Rain"].view()
../_images/70ddb90f8f243e30fda7657649658611f0f20051ab6cc4e4a6182eee659ca203.png

Learn collection-wise epochs

df_local_epochs = tsc.merge_local_epochs()
df_local_epochs
name id_epoch n_epoch n_gaps_small start end color
0 Rain 1 150 3 2019-01-01 00:00:00 2019-05-30 00:00:00 #1b9e77
1 Rain 2 195 0 2019-06-20 00:00:00 2019-12-31 00:00:00 #666666
2 Flow 1 275 2 2019-03-01 00:00:00 2019-11-30 00:00:00 #1b9e77
3 Flow2 1 256 5 2019-03-20 00:00:00 2019-11-30 00:00:00 #1b9e77
4 AirTemp 1 2000 5 2019-01-01 00:00:00 2019-03-25 07:00:00 #1b9e77
5 AirTemp 2 2249 0 2019-03-28 08:00:00 2019-06-30 00:00:00 #666666
6 ET 1 60 0 2019-01-01 00:00:00 2019-03-01 00:00:00 #1b9e77
7 ET 2 230 0 2019-03-12 00:00:00 2019-10-27 00:00:00 #66a61e
8 ET 3 50 0 2019-11-12 00:00:00 2019-12-31 00:00:00 #666666

Visualize collection-wise epochs

tsc.view()
../_images/1ad174fec15a6eec72f45c4e792de1bae9e844d87b0d2a9d643026f9c5283aa3.png

Get merged data epochs

tsc.get_epochs()
datetime v_T1 v_ET1 v_Q1 v_Q2 v_P1 Overlapping id_epoch
0 2019-01-01 00:00:00 26.870612 4.062957 NaN NaN 10.285795 0.6 1
1 2019-01-01 01:00:00 26.841576 NaN NaN NaN NaN 0.2 1
2 2019-01-01 02:00:00 27.269388 NaN NaN NaN NaN 0.2 1
3 2019-01-01 03:00:00 27.441897 NaN NaN NaN NaN 0.2 1
4 2019-01-01 04:00:00 27.040871 NaN NaN NaN NaN 0.2 1
... ... ... ... ... ... ... ... ...
4500 2019-12-27 00:00:00 NaN 4.598460 NaN NaN 10.239898 0.4 5
4501 2019-12-28 00:00:00 NaN 4.682095 NaN NaN 11.556343 0.4 5
4502 2019-12-29 00:00:00 NaN 4.329283 NaN NaN 8.844813 0.4 5
4503 2019-12-30 00:00:00 NaN 4.136005 NaN NaN 9.161709 0.4 5
4504 2019-12-31 00:00:00 NaN 4.438775 NaN NaN 9.484644 0.4 5

4505 rows × 8 columns

Standardize collection#

Regularizes each series onto its own regular time step. Gaps stay as they are, since filling them is a separate decision (interpolate_gaps):

tsc.standardize()

for name in tsc.collection:
    d = tsc.collection[name]
    print(f"{name:>10}: n={len(d.data):5d}  nan={d.data['v'].isna().sum():5d}  "
          f"freq={d.dtfreq:>3}  start={d.start.date()}  end={d.end.date()}")

      Rain: n=  365  nan=   23  freq=  D  start=2019-01-01  end=2019-12-31
      Flow: n=  275  nan=    2  freq=  D  start=2019-03-01  end=2019-11-30
     Flow2: n=  256  nan=    5  freq=  D  start=2019-03-20  end=2019-11-30
   AirTemp: n= 4344  nan=  100  freq=  h  start=2019-01-01  end=2019-06-30
        ET: n=  365  nan=   25  freq=  D  start=2019-01-01  end=2019-12-31

Merge data#

The .merge_data() is a helper method that makes an outer join for the available time series, returning a dataframe with all the data.

df_merged = tsc.merge_data()
df_merged
datetime v_T1 v_ET1 v_Q1 v_Q2 v_P1
0 2019-01-01 00:00:00 26.870612 4.062957 NaN NaN 10.285795
1 2019-01-01 01:00:00 26.841576 NaN NaN NaN NaN
2 2019-01-01 02:00:00 27.269388 NaN NaN NaN NaN
3 2019-01-01 03:00:00 27.441897 NaN NaN NaN NaN
4 2019-01-01 04:00:00 27.040871 NaN NaN NaN NaN
... ... ... ... ... ... ...
4523 2019-12-27 00:00:00 NaN 4.598460 NaN NaN 10.239898
4524 2019-12-28 00:00:00 NaN 4.682095 NaN NaN 11.556343
4525 2019-12-29 00:00:00 NaN 4.329283 NaN NaN 8.844813
4526 2019-12-30 00:00:00 NaN 4.136005 NaN NaN 9.161709
4527 2019-12-31 00:00:00 NaN 4.438775 NaN NaN 9.484644

4528 rows × 6 columns

Recap#

  • Built 5 synthetic series with independently varied range, gap structure, and time step.

  • TimeSeriesCollection.load_data() reads them all from one info table into a shared catalog.

  • get_epochs() / merge_local_epochs() give the cross-series and per-series gap/epoch pictures respectively.