FitModelOverNestedFoldsUnit
Score every outer fold with a search that never saw it.
FitModelOverFoldsUnit plus one measurement taken before it. The
relationship is inheritance rather than a shared mixin because that is what
it is: everything the sibling does still happens, and this adds a step in
front of it.
What the extra step is for. In an ordinary cross-validated search, the
same folds choose the hyperparameters and report the score, so the reported
score is optimistic by however much the search managed to fit them. The
nested loop measures that honestly: for each outer fold, a search is run
from scratch on folds carved out of that fold's training rows only, and
the chosen model is then scored on the outer fold's validation rows, which
that search never saw. Those are the OUTER_FOLD rows.
What it does not do is choose the hyperparameters. Each outer fold picks its own, and they generally differ; there is no single model to keep out of that loop. So the ordinary search still runs afterwards, over all the folds, and it is what produces the model that gets saved. The nested numbers are a statement about the procedure, not about the artifact.
That also means the inner trials record nothing: their partitions belong to one outer fold, and rows from them would sit alongside rows describing the kept model as if they were comparable.
Parameters
- optimizer : object
- Optimizer used to search for hyperparameters, along with its own configuration. Only used when the model declares optimizable parameters.
- goal_metric : string, default=
Accuracy - Metric the hyperparameter search optimizes.
- scored_splits : array, default=
['TRAIN', 'VALIDATION'] - Partitions each trial of the search records a score for. A partition the model is not meant to be judged on belongs out of this list even when metrics are configured for it.
- inner_splitter : object
- Splitter that decides how the dataset is partitioned, along with its own configuration.
Methods
execute(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
FitModelOverNestedFoldsUnitDo the unit's work, reading from and writing to the context.
Parameters
- ctx : ExecutionContext
- The shared execution context.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
validate(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
FitModelOverFoldsUnitCheck preconditions without executing the unit.
Parameters
- ctx : ExecutionContext
- The shared execution context.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.