OpusMtEnRoaTranslator

Converter

DashAI.back.converters.hugging_face.translation.OpusMtEnRoaTranslator

English to a Romance language (Helsinki-NLP/opus-mt-en-roa).

Parameters

target_language : string, default=por: Language to translate the text into.
split_by : string, default=whole: Translate each value whole, or in parts that keep their line breaks and spacing. Parts help when one value holds a lot of text, such as a whole file loaded as a single document item; values already split by structure do not need them. A part too long for the model is always cut further, by lines, sentences and words, so no text is dropped.
batch_size : integer, default=8: Number of texts translated together. Lower it if memory runs out.
device : string, default=CPU: Device the model runs on. A GPU is much faster than the CPU.

Methods

build_report(self, data: 'Dict[str, Any]') -> 'Dict[str, Any]'

Defined on BaseConverter

Wrap converter-specific data with the producer converter name.

component_dir(cls) -> pathlib.Path

Defined on DownloadableMixin

Return this component's own storage directory.

Returns

pathlib.Path: <COMPONENT_PATH>/<ClassName>.

convert_text(self, value: str) -> str

Defined on TranslationConverter

Translate one value.

Parameters

value : str: The value.

Returns

str: Its translation.

convert_texts(self, values: List[Optional[str]]) -> List[Optional[str]]

Defined on TranslationConverter

Translate the values of one column in a single model run.

Parameters

values : List[Optional[str]]: The column values.

Returns

List[Optional[str]]: The translations; missing values and values without letters are kept.

count_tokens(self, text: str) -> int

Defined on TranslationConverter

Count the tokens the model reads for a text.

Parameters

text : str: The text.

Returns

int: Its token count, without special tokens.

delete(cls) -> None

Defined on DownloadableMixin

Remove the component's downloaded artifacts.

download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None

Defined on HFDownloadableMixin

Download all repos listed in hf_repos() into component_dir().

Parameters

report : ProgressReporter, optional: Callback invoked before each repo download with report(None, "Downloading <repo_id>"). None means no progress reporting.

fit(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'TextValueConverter'

Defined on TextValueConverter

Record which selected columns are Text.

Parameters

x : DashAIDataset: The selected columns.
y : DashAIDataset, optional: Ignored.

Returns

TextValueConverter: The fitted converter.

generate_options(self, tokenizer: Any) -> Dict[str, Any]

Defined on TranslationConverter

Return extra generate arguments, such as the target language.

Parameters

tokenizer : Any: The loaded tokenizer.

Returns

Dict[str, Any]: Empty by default.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str: Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential: An instance of the requested credential component.

get_metadata(cls) -> 'Dict[str, Any]'

Defined on BaseConverter

Get metadata for the converter, used by the DashAI frontend.

Parameters

cls : type: The converter class (injected automatically by Python for classmethods).

Returns

Dict[str, Any]: Dictionary containing display name, short description, image preview path, category, icon, color, and whether the converter is supervised.

get_output_type(self, column_name: Optional[str] = None) -> DashAI.back.types.dashai_data_type.DashAIDataType

Defined on TextValueConverter

Return Text, the type of every rewritten column.

Parameters

column_name : str, optional: Unused; every rewritten column is Text.

Returns

DashAIDataType: A string backed Text type.

get_report(self) -> 'Union[Dict[str, Any], None]'

Defined on BaseConverter

Return the converter report produced after execution, if any.

Returns

Dict[str, Any] or None: JSON-serializable report dict, or None when the converter does not produce one.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict: Dictionary representing the Json Schema of the component.

hf_repos(cls)

Defined on HFPretrainedDownloadMixin

Derive the single repo entry from MODEL_NAME.

Returns

list of tuple of (str, str): [(MODEL_NAME, "model")] or an empty list when unset.

infer_output_columns(self, inputs: 'List[StateItem]') -> 'StructureDelta'

Defined on BaseConverter

Estimate what this converter does to its scope, without any data.

Parameters

inputs : list of ColumnItem | BlockItem: The dataset state items in this converter's scope.

Returns

StructureDelta: The scope items that survive (maybe retyped) and the new items.

is_downloaded(cls) -> bool

Defined on HFDownloadableMixin

Return whether all repo directories exist and are non-empty.

Returns

bool: True when every repo listed in hf_repos() has a non-empty local directory; False otherwise (including when the list is empty).

load_pretrained(self, source: str) -> Tuple[Any, Any]

Defined on TranslationConverter

Load the tokenizer and the sequence to sequence model.

Parameters

source : str: A local directory or the Hub repo id.

Returns

Tuple[Any, Any]: The tokenizer and the model, in evaluation mode on the device.

prepare_tokenizer(self, tokenizer: Any) -> None

Defined on TranslationConverter

Set up the tokenizer for the source language.

Parameters

tokenizer : Any: The loaded tokenizer.

pretrained(self) -> Any

Defined on PretrainedTool

Return the loaded model, loading it on first use.

Returns

Any: The value :meth:load_pretrained returned.

rows_to_keep(self, x: "'DashAIDataset'") -> 'List[int]'

Defined on BaseConverter

Positions of the rows of x that survive this converter.

Parameters

x : DashAIDataset: The converter's scope columns of one split.

Returns

list of int: 0-based positions of the kept rows, in order.

setup_device(self, label: Optional[str]) -> None

Defined on PretrainedTool

Choose the device and forget any loaded model.

Parameters

label : str, optional: The device parameter.

source_prefix(self) -> str

Defined on OpusMtMultiTargetTranslator

Return the token that selects the target language.

Returns

str: Such as ">>por<< ".

transform(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DashAIDataset'

Defined on TextValueConverter

Rewrite the values of the fitted Text columns.

Parameters

x : DashAIDataset: The dataset to transform.
y : DashAIDataset, optional: Ignored.

Returns

DashAIDataset: The dataset with every Text column rewritten.

translate_pieces(self, pieces: List[str]) -> List[str]

Defined on TranslationConverter

Translate short texts, each fitting the model input.

Parameters

pieces : List[str]: The texts.

Returns

List[str]: One translation per text, in the same order.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict: A dictionary with the data provided by the user to initialize the model.

Returns

dict: A validated dictionary with the necessary objects.

Parameters​

Methods​

Parameters

Methods