ForecastingHoldoutEvaluationStrategy
Holdout evaluation that records no in-sample metrics.
One thing the ordinary holdout strategy assumes is wrong for a forecaster, and it is a decision about evaluation rather than about any model.
The training partition is not scored. Scoring it would mean asking the model about dates it was fitted on. That is an in-sample fit statistic, which is a real diagnostic but is not comparable with a forecast made several steps out; showing the two side by side in one results table invites exactly that comparison. Only validation and test are recorded.
The kept model is fitted on the training partition alone, like every other holdout run, and nothing is fed to it afterwards. Two approaches that would have changed that were tried and dropped, both because they hand the model data from a partition it was meant to be held out from:
refitting through validation before scoring test, which overwrote the fit the validation metrics came from, so the saved model could not reproduce its own results table;
advancing the model through the observed validation rows at predict time, which re-estimates nothing but still lets a held out partition reach the model, which no other task in DashAI does.
So the two columns describe different horizons, and deliberately:
validation metrics <- forecasting 1..len(val) past the fit test metrics <- forecasting len(val)+1..len(val)+len(test), its own forecasts standing in for validation
The test column is therefore the harder question, not the same one further
along. Comparing like with like over a chosen horizon is what
RollingOriginSplitter is for, since its horizon says outright how
many steps ahead each refit is scored on.
Hyperparameter search is untouched. Its trials are scored on validation, so they must not be fitted on it.
Methods
evaluate(self, model, input_dataset, output_dataset, metric)
SinglePartitionEvaluationStrategyEvaluate model on validation set during HPO trials.
Parameters
- model : BaseModel
- The model instance to evaluate with specific hyperparameters.
- input_dataset : DatasetDict
- DatasetDict with data partitions {"train": X_train, "validation": X_val, "test": X_test}.
- output_dataset : DatasetDict
- DatasetDict with label partitions {"train": y_train, "validation": y_val, "test": y_test}.
- metric : Metric
- The metric function to compute on predictions.
Returns
- float
- The metric score value for this hyperparameter combination.
execute(self, x, y, run: DashAI.back.dependencies.database.models.Run, db)
SinglePartitionEvaluationStrategyExecute holdout validation: train on training set, optimize with validation, evaluate on test.
Parameters
- x : DatasetDict
- DatasetDict with data partitions: {"train": X_train, "validation": X_val, "test": X_test}
- y : DatasetDict
- DatasetDict with label partitions: {"train": y_train, "validation": y_val, "test": y_test}
- run : Run
- Database run instance for storing results and configuration.
- db : Session
- SQLAlchemy database session for persisting metrics.
Returns
- tuple
- (trained_model, plot_paths) where: - trained_model : BaseModel - The trained model - plot_paths : list[str] - Paths to HPO visualization plot files
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds, andscored_splits, the partitions it writes metrics for. A screen that offers one control per partition reads the latter instead of assuming all three exist: a forecasting strategy scores no training partition, so asking it for train metrics finds nothing.
set_progress_reporter(self, progress_reporter: Optional[Callable[[Optional[float], Optional[str]], NoneType]]) -> None
BaseEvaluationStrategyRegister a callback that will receive progress updates.