ForecastingHoldoutEvaluationStrategy
A backtest with one validation origin, then the test.
Pair it with TemporalHoldoutSplitter. The model is fitted on the steps
before the validation window and scored on its horizon steps, then
refitted on everything before the test and scored on the last horizon
steps. Validation and test measure the same horizon, so the two columns of
the results compare like with like.
It runs the same units as the rolling origin (KIND = "backtest"): a
holdout is a backtest with one origin. The training steps are never
scored, since a forecaster asked about steps it was fitted on reports a
fit, not a forecast. A search measures its trials on validation only. The
kept model is fitted on the whole series, and the metrics are the
backtest's (RECONCILING_FORECASTING §2.13).
Methods
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds;scored_splits, the partitions it writes metrics for;splitter_partitioninganddefault_splitter, which splitters it takes and which one a session starts on;splitters, the names it is limited to, or None; andresult_views, the views a finished run offers.