LanguageFilter

Converter

DashAI.back.converters.document.LanguageFilter

---

Drop the selected Text items written in other languages.

The language of each item is detected with papluca/xlm-roberta-base-language-detection, which knows 20 languages and reads the first 256 tokens of an item.

Parameters

languages : array, default=['en']

Language codes of the items to keep, such as 'en' or 'es'. Known codes: ar, bg, de, el, en, es, fr, hi, it, ja, nl, pl, pt, ru, sw, th, tr, ur, vi, zh.

min_confidence : number, default=0.5

Lowest probability, between 0 and 1, for a detected language to count. Below it the language is uncertain.

keep_uncertain : boolean, default=True

Keep items whose language is uncertain, such as numbers, names or very short text.

device : string, default=CPU

Device the model runs on. A GPU is much faster than the CPU.

Methods

detect(self, values: List[str]) -> List[Tuple[str, float]]

Defined on LanguageFilter

Detect the most likely language of each text.

Parameters

values : List[str]

The texts.

Returns

List[Tuple[str, float]]

(code, probability) for each text.

load_pretrained(self, source: str) -> Tuple[Any, Any]

Defined on LanguageFilter

Load the tokenizer and the classifier.

Parameters

source : str

A local directory or the Hub repo id.

Returns

Tuple[Any, Any]

The tokenizer and the model, in evaluation mode on the device.

transform(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DashAIDataset'

Defined on LanguageFilter

Return the items in the chosen languages.

Parameters

x : DashAIDataset

The selected Text items, in one column.

y : DashAIDataset, optional

Ignored.

Returns

DashAIDataset

The kept items with their source rows.

build_report(self, data: 'Dict[str, Any]') -> 'Dict[str, Any]'

Defined on BaseConverter

Wrap converter-specific data with the producer converter name.

component_dir(cls) -> pathlib.Path

Defined on DownloadableMixin

Return this component's own storage directory.

Returns

pathlib.Path

<COMPONENT_PATH>/<ClassName>.

delete(cls) -> None

Defined on DownloadableMixin

Remove the component's downloaded artifacts.

download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None

Defined on HFDownloadableMixin

Download all repos listed in hf_repos() into component_dir().

Parameters

report : ProgressReporter, optional

Callback invoked before each repo download with report(None, "Downloading <repo_id>"). None means no progress reporting.

fit(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DocumentConverter'

Defined on DocumentConverter

Return the converter unchanged; override to learn from the items.

Parameters

x : DashAIDataset

The selected items.

y : DashAIDataset, optional

Ignored.

Returns

DocumentConverter

This converter.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str

Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential

An instance of the requested credential component.

get_metadata(cls) -> 'Dict[str, Any]'

Defined on BaseConverter

Get metadata for the converter, used by the DashAI frontend.

Parameters

cls : type

The converter class (injected automatically by Python for classmethods).

Returns

Dict[str, Any]

Dictionary containing display name, short description, image preview path, category, icon, color, and whether the converter is supervised.

get_output_type(self, column_name: Optional[str] = None) -> DashAI.back.types.dashai_data_type.DashAIDataType

Defined on DocumentConverter

Return Text, the content type most document converters produce.

Parameters

column_name : str, optional

Unused.

Returns

DashAIDataType

A string backed Text type.

get_report(self) -> 'Union[Dict[str, Any], None]'

Defined on BaseConverter

Return the converter report produced after execution, if any.

Returns

Dict[str, Any] or None

JSON-serializable report dict, or None when the converter does not produce one.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict

Dictionary representing the Json Schema of the component.

hf_repos(cls)

Defined on HFPretrainedDownloadMixin

Derive the single repo entry from MODEL_NAME.

Returns

list of tuple of (str, str)

[(MODEL_NAME, "model")] or an empty list when unset.

infer_output_columns(self, inputs: 'List[StateItem]') -> 'StructureDelta'

Defined on BaseConverter

Estimate what this converter does to its scope, without any data.

Parameters

inputs : list of ColumnItem | BlockItem

The dataset state items in this converter's scope.

Returns

StructureDelta

The scope items that survive (maybe retyped) and the new items.

is_downloaded(cls) -> bool

Defined on HFDownloadableMixin

Return whether all repo directories exist and are non-empty.

Returns

bool

True when every repo listed in hf_repos() has a non-empty local directory; False otherwise (including when the list is empty).

items_result(contents: List[Any], sources: List[int], content_type: type = <class 'DashAI.back.types.value_types.Text'>, kinds: Optional[List[Optional[str]]] = None, attrs: Optional[List[Optional[Dict[str, Any]]]] = None) -> 'DashAIDataset'

Defined on DocumentConverter

Build the dataset a document converter returns.

Parameters

contents : List[Any]

The new item contents: strings for Text, {"bytes", "path"} dicts for images.

sources : List[int]

For each new item, the index of the input row it comes from.

content_type : type, optional

Text (default) or DashAIImage.

kinds : List[Optional[str]], optional

The element kind of each new item; None inherits or defaults.

attrs : List[Optional[Dict[str, Any]]], optional

The element attributes of each new item.

Returns

DashAIDataset

content and source columns, plus kind and attrs when given.

pretrained(self) -> Any

Defined on PretrainedTool

Return the loaded model, loading it on first use.

Returns

Any

The value :meth:load_pretrained returned.

rows_to_keep(self, x: "'DashAIDataset'") -> 'List[int]'

Defined on BaseConverter

Positions of the rows of x that survive this converter.

Parameters

x : DashAIDataset

The converter's scope columns of one split.

Returns

list of int

0-based positions of the kept rows, in order.

setup_device(self, label: Optional[str]) -> None

Defined on PretrainedTool

Choose the device and forget any loaded model.

Parameters

label : str, optional

The device parameter.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict

A dictionary with the data provided by the user to initialize the model.

Returns

dict

A validated dictionary with the necessary objects.