HoldoutEvaluationStrategy
DashAI.back.evaluation.HoldoutEvaluationStrategy
Split once into train, validation and test, and score all three.
The ordinary holdout evaluation. Not offered for ForecastingTask:
scoring the training partition of a forecaster means predicting on dates
it was fitted on, which is a fit statistic rather than a forecast and does
not belong in the same results table as one.
ForecastingHoldoutEvaluationStrategy records validation and test only.
The kept model is fitted on the training partition alone in both, so it is the model the recorded metrics describe. The validation partition is handed to the fit, which models use to watch training and stop early -- being given it to watch is not the same as being fitted on it.
Methods
get_metadata(cls) -> dict
BaseEvaluationStrategyDescribe the strategy for the frontend.
Returns
- dict
- Mapping with
kind, which says whether this strategy splits the dataset once or into folds, andscored_splits, the partitions it writes metrics for.