ComputeDatasetMetadataUnit
Fill in the metadata a dataset carries in its splits mapping.
Two independent knobs, because "how much metadata" and "can the metadata already there be trusted" are different questions:
compute_metadatapicks the depth: base only (column names, row count, NaN counts) or base plus the extended exploratory fields.trust_inherited_metadatasays the dataset arrived carrying metadata that still describes it — the case of a copy whose bytes never changed. Then the unit fills only what is missing instead of recomputing.
Either way the requested depth is what ends up stored: asking for base only always strips the extended keys, even ones inherited from elsewhere, so the result never depends on where the dataset came from.
Re-publishes dataset although it mutates it in place: the key is what
makes the unit chainable, and the contract audit reads the ctx.put call.
Parameters
- compute_metadata : boolean, default=
True - Whether to compute the extended exploratory metadata (per-column statistics, quality report, correlations) on top of the base one. Disabling it makes large datasets much faster to store, at the cost of the exploration views.
Methods
execute(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
ComputeDatasetMetadataUnitDo the unit's work, reading from and writing to the context.
Parameters
- ctx : ExecutionContext
- The shared execution context.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
validate(self, ctx: DashAI.back.units.context.ExecutionContext) -> None
BaseUnitCheck preconditions without executing the unit.
Parameters
- ctx : ExecutionContext
- The shared execution context.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.