ForecastingHoldoutEvaluationStrategy
Holdout evaluation that treats validation as history rather than a sample.
Two things the ordinary holdout strategy assumes are wrong for a forecaster, and both of them are decisions about evaluation rather than about any model.
The training partition is not scored. Scoring it would mean asking the model about dates it was fitted on. That is an in-sample fit statistic, which is a real diagnostic but is not comparable with a forecast made several steps out; showing the two side by side in one results table invites exactly that comparison. Only validation and test are recorded.
The kept model is fitted through validation. For most tasks the validation partition is a held out sample that has to stay out of the fit. For a forecaster it is simply the most recent stretch of the series, and the stretch nearest to whatever comes next. Leaving it out makes the model reach across the whole validation window before arriving at the first test row, so the test metrics describe a longer horizon than the one being asked about.
The validation metrics are still measured on a model fitted on training data alone, which is what makes them honest: they are recorded before the refit. So the two columns in the results table answer different questions, and both answer them fairly.
validation metrics <- model fitted on train test metrics <- model fitted on train + validation
Hyperparameter search is untouched. Its trials are scored on validation, so they must not be fitted on it.
Methods
execute(self, x, y, run, db)
ForecastingHoldoutEvaluationStrategyScore validation on a trial fit, then refit and score test.
Parameters
- x : DatasetDict
- Input partitions, keyed by split name.
- y : DatasetDict
- Target partitions, keyed by split name.
- run : Run
- Database model representing the current run.
- db : Session
- SQLAlchemy session used to persist metrics.
Returns
- tuple
- The trained model and the paths of any HPO plots.
evaluate(self, model, input_dataset, output_dataset, metric)
SinglePartitionEvaluationStrategyEvaluate model on validation set during HPO trials.
Parameters
- model : BaseModel
- The model instance to evaluate with specific hyperparameters.
- input_dataset : DatasetDict
- DatasetDict with data partitions {"train": X_train, "validation": X_val, "test": X_test}.
- output_dataset : DatasetDict
- DatasetDict with label partitions {"train": y_train, "validation": y_val, "test": y_test}.
- metric : Metric
- The metric function to compute on predictions.
Returns
- float
- The metric score value for this hyperparameter combination.
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds.
set_progress_reporter(self, progress_reporter: Optional[Callable[[Optional[float], Optional[str]], NoneType]]) -> None
BaseEvaluationStrategyRegister a callback that will receive progress updates.