Summarizer
Replace each long text value with an abstractive summary.
Uses csebuetnlp/mT5_multilingual_XLSum, trained on news in 45
languages, which writes the summary in the language of the text and
reads up to 512 tokens. Values shorter than min_words are kept.
Parameters
- min_words : integer, default=
60 - Values with fewer words are kept as they are; summarizing a short text does not shorten it.
- max_summary_tokens : integer, default=
84 - Longest summary, in model tokens (about 3/4 of a word each).
- device : string, default=
CPU - Device the model runs on. A GPU is much faster than the CPU.
Methods
convert_text(self, value: str) -> str
SummarizerSummarize one value when it is long enough.
Parameters
- value : str
- The value.
Returns
- str
- Its summary, or the value itself when it is short.
convert_texts(self, values: List[Optional[str]]) -> List[Optional[str]]
SummarizerSummarize the long values of one column and keep the others.
Parameters
- values : List[Optional[str]]
- The column values.
Returns
- List[Optional[str]]
- Summaries for the values with at least
min_wordswords; a value whose summary comes out empty is kept.
load_pretrained(self, source: str) -> Tuple[Any, Any]
SummarizerLoad the tokenizer and the sequence to sequence model.
Parameters
- source : str
- A local directory or the Hub repo id.
Returns
- Tuple[Any, Any]
- The tokenizer and the model, in evaluation mode on the device.
summarize(self, values: List[str]) -> List[str]
SummarizerSummarize texts.
Parameters
- values : List[str]
- The texts.
Returns
- List[str]
- One summary per text.
build_report(self, data: 'Dict[str, Any]') -> 'Dict[str, Any]'
BaseConverterWrap converter-specific data with the producer converter name.
component_dir(cls) -> pathlib.Path
DownloadableMixinReturn this component's own storage directory.
Returns
- pathlib.Path
<COMPONENT_PATH>/<ClassName>.
delete(cls) -> None
DownloadableMixinRemove the component's downloaded artifacts.
download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None
HFDownloadableMixinDownload all repos listed in hf_repos() into component_dir().
Parameters
- report : ProgressReporter, optional
- Callback invoked before each repo download with
report(None, "Downloading <repo_id>").Nonemeans no progress reporting.
fit(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'TextValueConverter'
TextValueConverterRecord which selected columns are Text.
Parameters
- x : DashAIDataset
- The selected columns.
- y : DashAIDataset, optional
- Ignored.
Returns
- TextValueConverter
- The fitted converter.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_metadata(cls) -> 'Dict[str, Any]'
BaseConverterGet metadata for the converter, used by the DashAI frontend.
Parameters
- cls : type
- The converter class (injected automatically by Python for classmethods).
Returns
- Dict[str, Any]
- Dictionary containing display name, short description, image preview path, category, icon, color, and whether the converter is supervised.
get_output_type(self, column_name: Optional[str] = None) -> DashAI.back.types.dashai_data_type.DashAIDataType
TextValueConverterReturn Text, the type of every rewritten column.
Parameters
- column_name : str, optional
- Unused; every rewritten column is Text.
Returns
- DashAIDataType
- A string backed
Texttype.
get_report(self) -> 'Union[Dict[str, Any], None]'
BaseConverterReturn the converter report produced after execution, if any.
Returns
- Dict[str, Any] or None
- JSON-serializable report dict, or
Nonewhen the converter does not produce one.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
hf_repos(cls)
HFPretrainedDownloadMixinDerive the single repo entry from MODEL_NAME.
Returns
- list of tuple of (str, str)
[(MODEL_NAME, "model")]or an empty list when unset.
infer_output_columns(self, inputs: 'List[StateItem]') -> 'StructureDelta'
BaseConverterEstimate what this converter does to its scope, without any data.
Parameters
- inputs : list of ColumnItem | BlockItem
- The dataset state items in this converter's scope.
Returns
- StructureDelta
- The scope items that survive (maybe retyped) and the new items.
is_downloaded(cls) -> bool
HFDownloadableMixinReturn whether all repo directories exist and are non-empty.
Returns
- bool
Truewhen every repo listed inhf_repos()has a non-empty local directory;Falseotherwise (including when the list is empty).
pretrained(self) -> Any
PretrainedToolReturn the loaded model, loading it on first use.
Returns
- Any
- The value :meth:
load_pretrainedreturned.
rows_to_keep(self, x: "'DashAIDataset'") -> 'List[int]'
BaseConverterPositions of the rows of x that survive this converter.
Parameters
- x : DashAIDataset
- The converter's scope columns of one split.
Returns
- list of int
- 0-based positions of the kept rows, in order.
setup_device(self, label: Optional[str]) -> None
PretrainedToolChoose the device and forget any loaded model.
Parameters
- label : str, optional
- The
deviceparameter.
transform(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DashAIDataset'
TextValueConverterRewrite the values of the fitted Text columns.
Parameters
- x : DashAIDataset
- The dataset to transform.
- y : DashAIDataset, optional
- Ignored.
Returns
- DashAIDataset
- The dataset with every Text column rewritten.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.