TextCleaner

Converter

DashAI.back.converters.simple_converters.TextCleaner

---

Clean the artifacts text extraction leaves in text values.

Each step can be turned off: Unicode NFKC normalization, rejoining words hyphenated across a line break, removing invisible control and format characters, and collapsing whitespace while keeping line breaks.

Parameters

normalize_unicode : boolean, default=True

Apply Unicode NFKC normalization: ligatures such as 'fi' become 'fi' and full width or styled characters become plain ones.

join_hyphenated : boolean, default=True

Rejoin words split by a hyphen at the end of a line, as in 'exam-' and 'ple' on the next line.

remove_control_characters : boolean, default=True

Remove invisible control and format characters, such as zero width spaces and soft hyphens. Line breaks and tabs are kept.

collapse_whitespace : boolean, default=True

Turn runs of spaces into one space, remove spaces around line breaks, keep at most one blank line, and trim the value.

Methods

convert_text(self, value: str) -> str

Defined on TextCleaner

Clean one text value.

Parameters

value : str

The value to clean.

Returns

str

The cleaned value.

build_report(self, data: 'Dict[str, Any]') -> 'Dict[str, Any]'

Defined on BaseConverter

Wrap converter-specific data with the producer converter name.

convert_texts(self, values: List[Optional[str]]) -> List[Optional[str]]

Defined on TextValueConverter

Rewrite the values of one column, keeping missing values.

Parameters

values : List[Optional[str]]

The column values.

Returns

List[Optional[str]]

The rewritten values, None where the input was None.

fit(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'TextValueConverter'

Defined on TextValueConverter

Record which selected columns are Text.

Parameters

x : DashAIDataset

The selected columns.

y : DashAIDataset, optional

Ignored.

Returns

TextValueConverter

The fitted converter.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str

Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential

An instance of the requested credential component.

get_metadata(cls) -> 'Dict[str, Any]'

Defined on BaseConverter

Get metadata for the converter, used by the DashAI frontend.

Parameters

cls : type

The converter class (injected automatically by Python for classmethods).

Returns

Dict[str, Any]

Dictionary containing display name, short description, image preview path, category, icon, color, and whether the converter is supervised.

get_output_type(self, column_name: Optional[str] = None) -> DashAI.back.types.dashai_data_type.DashAIDataType

Defined on TextValueConverter

Return Text, the type of every rewritten column.

Parameters

column_name : str, optional

Unused; every rewritten column is Text.

Returns

DashAIDataType

A string backed Text type.

get_report(self) -> 'Union[Dict[str, Any], None]'

Defined on BaseConverter

Return the converter report produced after execution, if any.

Returns

Dict[str, Any] or None

JSON-serializable report dict, or None when the converter does not produce one.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict

Dictionary representing the Json Schema of the component.

infer_output_columns(self, inputs: 'List[StateItem]') -> 'StructureDelta'

Defined on BaseConverter

Estimate what this converter does to its scope, without any data.

Parameters

inputs : list of ColumnItem | BlockItem

The dataset state items in this converter's scope.

Returns

StructureDelta

The scope items that survive (maybe retyped) and the new items.

rows_to_keep(self, x: "'DashAIDataset'") -> 'List[int]'

Defined on BaseConverter

Positions of the rows of x that survive this converter.

Parameters

x : DashAIDataset

The converter's scope columns of one split.

Returns

list of int

0-based positions of the kept rows, in order.

transform(self, x: 'DashAIDataset', y: Optional[ForwardRef('DashAIDataset')] = None) -> 'DashAIDataset'

Defined on TextValueConverter

Rewrite the values of the fitted Text columns.

Parameters

x : DashAIDataset

The dataset to transform.

y : DashAIDataset, optional

Ignored.

Returns

DashAIDataset

The dataset with every Text column rewritten.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict

A dictionary with the data provided by the user to initialize the model.

Returns

dict

A validated dictionary with the necessary objects.