HoldoutEvaluationStrategy
Split once into train, validation and test, and score all three.
The ordinary holdout evaluation. Not offered for ForecastingTask:
scoring the training partition of a forecaster means predicting on dates
it was fitted on, which is a fit statistic rather than a forecast and does
not belong in the same results table as one.
ForecastingHoldoutEvaluationStrategy records validation and test only.
The final fit is the same in both: the kept model is fitted on the training partition alone, so it is the model the recorded metrics describe.
Methods
evaluate(self, model, input_dataset, output_dataset, metric)
SinglePartitionEvaluationStrategyEvaluate model on validation set during HPO trials.
Parameters
- model : BaseModel
- The model instance to evaluate with specific hyperparameters.
- input_dataset : DatasetDict
- DatasetDict with data partitions {"train": X_train, "validation": X_val, "test": X_test}.
- output_dataset : DatasetDict
- DatasetDict with label partitions {"train": y_train, "validation": y_val, "test": y_test}.
- metric : Metric
- The metric function to compute on predictions.
Returns
- float
- The metric score value for this hyperparameter combination.
execute(self, x, y, run: DashAI.back.dependencies.database.models.Run, db)
SinglePartitionEvaluationStrategyExecute holdout validation: train on training set, optimize with validation, evaluate on test.
Parameters
- x : DatasetDict
- DatasetDict with data partitions: {"train": X_train, "validation": X_val, "test": X_test}
- y : DatasetDict
- DatasetDict with label partitions: {"train": y_train, "validation": y_val, "test": y_test}
- run : Run
- Database run instance for storing results and configuration.
- db : Session
- SQLAlchemy database session for persisting metrics.
Returns
- tuple
- (trained_model, plot_paths) where: - trained_model : BaseModel - The trained model - plot_paths : list[str] - Paths to HPO visualization plot files
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds, andscored_splits, the partitions it writes metrics for. A screen that offers one control per partition reads the latter instead of assuming all three exist: a forecasting strategy scores no training partition, so asking it for train metrics finds nothing.
set_progress_reporter(self, progress_reporter: Optional[Callable[[Optional[float], Optional[str]], NoneType]]) -> None
BaseEvaluationStrategyRegister a callback that will receive progress updates.