ForecastingCrossValidationEvaluationStrategy
Rolling origin cross-validation that records no in-sample metrics.
Only one thing separates this from the ordinary cross-validation strategy: the training partition of a fold is not scored. Scoring it would mean asking the model about dates it was fitted on, which is a fit statistic rather than a forecast and is not comparable with the validation score of the same fold.
Nothing else needs to change, and that is worth stating because it was not
obvious. Each fold already trains on everything before its own validation
window, and the final refit already uses the whole pool of rows outside
the reserved tail, so this strategy never had the horizon problem that
holdout did. Pair it with RollingOriginSplitter, whose folds walk the
origin forward through time.
Methods
evaluate(self, model, input_dataset, output_dataset, metric, **kwargs)
FoldEvaluationStrategyEvaluate model using k-fold cross-validation (used as HPO objective function).
Parameters
- model : BaseModel
- The model instance to evaluate (with specific hyperparameters).
- input_dataset : list of DatasetDict
- List of fold data {"train": X_train, "validation": X_validation}.
- output_dataset : list of DatasetDict
- List of fold labels {"train": y_train, "validation": y_validation}.
- metric : Metric
- The metric function to optimize.
- **kwargs
- Additional arguments including: - fold_index : int or None Inner outer fold index in nested CV (None for simple CV)
Returns
- float
- Mean metric value across all k folds (objective value for HPO).
- Note: When fold_index is provided (nested CV inner loop),
- intermediate metrics are NOT being saved (only outer loop metrics are saved).
execute(self, x, y, run: DashAI.back.dependencies.database.models.Run, db)
FoldEvaluationStrategyExecute k-fold cross-validation with optional nested CV and HPO.
Parameters
- x : list of DatasetDict
- List of fold DatasetDict each containing: {"train": X_train, "validation": X_validation} The last element is not a fold: it holds every row the folds could use as {"train": X_pool, "test": X_test}, where the test partition is empty when the session reserved nothing.
- y : list of DatasetDict
- List of fold label DatasetDicts with same structure as x.
- run : Run
- Database run instance containing configuration (nested CV settings, etc.).
- db : Session
- SQLAlchemy database session for persisting metrics and parameters.
Returns
- tuple
- (trained_model, plot_paths) where: - trained_model : BaseModel - The trained model - plot_paths : list[str] - Paths to HPO visualization plot files
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds.
set_progress_reporter(self, progress_reporter: Optional[Callable[[Optional[float], Optional[str]], NoneType]]) -> None
BaseEvaluationStrategyRegister a callback that will receive progress updates.