ScoreForecastsUnit
Score each origin's forecasts into the rows a run's metrics are.
Publishes forecast_metrics, a list of plain rows:
- one
FOLDrow per validation origin and metric, with the origin's index, as the folds of a cross-validation have theirs; - one
LASTvalidation row per metric, holding the mean over the origins and its standard deviation (0 for a single origin, as a holdout has); - one
LASTtest row per metric, for the test origin.
A metric with no value on an origin (UndefinedMetricError, or a score
that is not a number) has a None value and the reason, and then the
summary has none either, with the first origin's reason: averaging the
others would summarise a different set of origins.
The rows are values, not database rows: a unit does not write domain rows
(test_unit_contracts), so the job persists them. Unlike
cross-validation, where the job also computes the summary, the summary is
computed here with the scores it summarises, so it is addressable in a
graph (FORECASTING U9).
Parameters
- validation_metrics : array, default=
[] - Metrics scored on every validation origin, and their mean.
- test_metrics : array, default=
[] - Metrics scored on the test origin.
Methods
execute(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
ScoreForecastsUnitDo the unit's work, reading from and writing to the context.
Parameters
- ctx : ExecutionContext
- The shared execution context.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
validate(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
BaseUnitCheck preconditions without executing the unit.
Parameters
- ctx : ExecutionContext
- The shared execution context.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.