CrossValidationEvaluationStrategy
Score a model across folds, recording train and validation for each.
The ordinary cross-validation evaluation. Not offered for
ForecastingTask, whose folds have no in-sample score to report;
ForecastingCrossValidationEvaluationStrategy handles that.
Methods
evaluate(self, model, input_dataset, output_dataset, metric, **kwargs)
FoldEvaluationStrategyEvaluate model using k-fold cross-validation (used as HPO objective function).
Parameters
- model : BaseModel
- The model instance to evaluate (with specific hyperparameters).
- input_dataset : list of DatasetDict
- List of fold data {"train": X_train, "validation": X_validation}.
- output_dataset : list of DatasetDict
- List of fold labels {"train": y_train, "validation": y_validation}.
- metric : Metric
- The metric function to optimize.
- **kwargs
- Additional arguments including: - fold_index : int or None Inner outer fold index in nested CV (None for simple CV)
Returns
- float
- Mean metric value across all k folds (objective value for HPO).
- Note: When fold_index is provided (nested CV inner loop),
- intermediate metrics are NOT being saved (only outer loop metrics are saved).
execute(self, x, y, run: DashAI.back.dependencies.database.models.Run, db)
FoldEvaluationStrategyExecute k-fold cross-validation with optional nested CV and HPO.
Parameters
- x : list of DatasetDict
- List of fold DatasetDict each containing: {"train": X_train, "validation": X_validation} The last element is not a fold: it holds every row the folds could use as {"train": X_pool, "test": X_test}, where the test partition is empty when the session reserved nothing.
- y : list of DatasetDict
- List of fold label DatasetDicts with same structure as x.
- run : Run
- Database run instance containing configuration (nested CV settings, etc.).
- db : Session
- SQLAlchemy database session for persisting metrics and parameters.
Returns
- tuple
- (trained_model, plot_paths) where: - trained_model : BaseModel - The trained model - plot_paths : list[str] - Paths to HPO visualization plot files
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds, andscored_splits, the partitions it writes metrics for. A screen that offers one control per partition reads the latter instead of assuming all three exist: a forecasting strategy scores no training partition, so asking it for train metrics finds nothing.
set_progress_reporter(self, progress_reporter: Optional[Callable[[Optional[float], Optional[str]], NoneType]]) -> None
BaseEvaluationStrategyRegister a callback that will receive progress updates.