CrossValidationEvaluationStrategy

EvaluationStrategy

DashAI.back.evaluation.CrossValidationEvaluationStrategy

Evaluation strategy implementing k-fold cross-validation with optional nested CV and HPO.

This strategy partitions the dataset into k folds and performs k rounds of training and evaluation.

The strategy handles metric aggregation at multiple levels:

Methods

evaluate(self, model, input_dataset, output_dataset, metric, **kwargs)

Defined on CrossValidationEvaluationStrategy

Evaluate model using k-fold cross-validation (used as HPO objective function).

Parameters

model : BaseModel: The model instance to evaluate (with specific hyperparameters).
input_dataset : list of DatasetDict: List of fold data {"train": X_train, "test": X_test}.
output_dataset : list of DatasetDict: List of fold labels {"train": y_train, "test": y_test}.
metric : Metric: The metric function to optimize.
**kwargs: Additional arguments including: - fold_index : int or None Inner outer fold index in nested CV (None for simple CV)

Returns

float: Mean metric value across all k folds (objective value for HPO).
Note: When fold_index is provided (nested CV inner loop),
intermediate metrics are NOT being saved (only outer loop metrics are saved).

execute(self, x, y, run: DashAI.back.dependencies.database.models.Run, db)

Defined on CrossValidationEvaluationStrategy

Execute k-fold cross-validation with optional nested CV and HPO.

Parameters

x : list of DatasetDict: List of fold DatasetDict each containing: {"train": X_train, "test": X_test} The last element is the complete dataset for final training.
y : list of DatasetDict: List of fold label DatasetDicts with same structure as x.
run : Run: Database run instance containing configuration (nested CV settings, etc.).
db : Session: SQLAlchemy database session for persisting metrics and parameters.

Returns

tuple: (trained_model, plot_paths) where: - trained_model : BaseModel - The trained model - plot_paths : list[str] - Paths to HPO visualization plot files

set_progress_reporter(self, progress_reporter: Optional[Callable[[Optional[float], Optional[str]], NoneType]]) -> None

Defined on BaseEvaluationStrategy