EvaluateModelUnit
Compute the final metrics of a trained model, once per split.
The unit is idempotent: a split whose final metrics were already logged (for instance by a model that evaluates itself while training) is skipped.
Which metrics are computed is decided by the model, not by this unit:
ModelFactory attaches the metric classes to the model instance, and
BaseModel.calculate_metrics is final. That method persists the
rows through a session of its own, so these writes are not part of the
transaction the calling job controls.
This unit needs a real Run row and cannot be used without one.
Everything it does is write Metric rows against a foreign key to
run.id, so there is nothing left for it to do when there is no run.
validate refuses a missing run id rather than letting it through,
because the failure would otherwise be silent in both directions: the
idempotency query below would match no row whatever was already logged,
and calculate_metrics no-ops on a model with no run — so the unit
would report success having written nothing at all.
A caller that wants a model's metrics without a run wants a different unit, one that returns them instead of persisting them.
Parameters
- splits : array, default=
['TRAIN', 'VALIDATION', 'TEST'] - Data splits the model is evaluated on.
Methods
execute(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
EvaluateModelUnitDo the unit's work, reading from and writing to the context.
Parameters
- ctx : ExecutionContext
- The shared execution context.
validate(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
EvaluateModelUnitCheck preconditions without executing the unit.
Parameters
- ctx : ExecutionContext
- The shared execution context.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.