PlainTextDocumentDataLoader

DataLoader

DashAI.back.dataloaders.classes.PlainTextDocumentDataLoader

---

Parse a text file into paragraphs, or into one Text item for the file.

Parameters

encoding : string, default=utf-8

Character encoding the file is read with.

split_paragraphs : boolean, default=False

Store each paragraph, separated by blank lines, as its own text item. Turn it off to keep the whole file as a single text item.

Methods

load_document(self, file_path: str, params: Dict[str, Any]) -> 'DashAIDocument'

Defined on PlainTextDocumentDataLoader

Read a text file into paragraphs or into one text item.

Parameters

file_path : str

Path of the text file.

params : Dict[str, Any]

encoding and split_paragraphs; missing keys take their defaults.

Returns

DashAIDocument

One Text item per blank line separated paragraph when split_paragraphs is True, else one Text item with the whole stripped text; no item for a blank file; page=None.

build_metadata(self, file_path: str, params: Dict[str, Any], n_pages: Optional[int]) -> Dict[str, Any]

Defined on DocumentDataLoader

Build the metadata stored with the parsed document.

Parameters

file_path : str

Path of the parsed file.

params : Dict[str, Any]

The resolved loader parameters.

n_pages : Optional[int]

Number of pages, None when the format has no pages.

Returns

Dict[str, Any]

file_name, file_type, n_pages, dataloader, dataloader_params.

extract_files(self, file_path: str, temp_path: str) -> str

Defined on BaseDataLoader

Extract a ZIP archive into a subdirectory of temp_path.

Parameters

file_path : str

Path to the ZIP archive to extract.

temp_path : str

Base temporary directory; extraction target is <temp_path>/files/.

Returns

str

Path of the directory containing the extracted files (<temp_path>/files/).

extract_native_types(self, filepath_or_buffer: str, params: Dict[str, Any]) -> Optional[Dict[str, Dict[str, Any]]]

Defined on BaseDataLoader

Extract column types from the file's own metadata, if available.

Parameters

filepath_or_buffer : str

Path to the file already prepared on disk (post-ZIP extraction).

params : Dict[str, Any]

Dataloader parameters, same dict that load_preview receives.

Returns

Dict[str, Dict[str, Any]] | None

Column name -> DashAI type dict, or None if the format does not provide native types.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str

Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential

An instance of the requested credential component.

get_metadata(cls) -> Dict[str, Any]

Defined on BaseDataLoader

Return the registry metadata the frontend reads.

Returns

Dict[str, Any]

category, supported_extensions, tags (those extensions as upper case file formats), supports_native_types and data_kinds, the list of data kinds the loader produces.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict

Dictionary representing the Json Schema of the component.

load_data(self, filepath_or_buffer, temp_path, params, n_sample=None)

Defined on DocumentDataLoader

Refuse to build a dataset.

load_preview(self, filepath_or_buffer: str, params: Dict[str, Any], n_rows: int = 10) -> 'DataFrame'

Defined on BaseDataLoader

Load a preview of the dataset using streaming.

Parameters

filepath_or_buffer : str

Path to the file or buffer to load.

params : Dict[str, Any]

Parameters for loading the data.

n_rows : int, optional

Number of rows to preview. Default is 10.

Returns

DataFrame

A pandas DataFrame with the preview data.

prepare_files(self, file_path: str, temp_path: str) -> str

Defined on BaseDataLoader

Resolve a file path or URL into a local path suitable for loading.

Parameters

file_path : str

Path to a local file, a ZIP archive, or an HTTP(S) URL.

temp_path : str

Temporary directory used for extraction of ZIP or URL downloads.

Returns

tuple of (str, str)

(path, type_path) where type_path is "dir" for extracted archives/URLs or "file" for plain local files.

resolve_params(self, params: Optional[Dict[str, Any]]) -> Dict[str, Any]

Defined on DocumentDataLoader

Fill the parameters the caller left out and validate them.

Parameters

params : Optional[Dict[str, Any]]

Loader parameters as the form sends them.

Returns

Dict[str, Any]

Every schema field with its validated value.

supports_file(cls, file_name: str) -> bool

Defined on DocumentDataLoader

Tell whether the loader accepts a file by its extension.

Parameters

file_name : str

Name or path of the file.

Returns

bool

True when the lowercased suffix is in SUPPORTED_EXTENSIONS.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict

A dictionary with the data provided by the user to initialize the model.

Returns

dict

A validated dictionary with the necessary objects.

web_image(data: bytes, name: str) -> 'DashAIImage'

Defined on DocumentDataLoader

Wrap extracted image bytes, converting them to PNG when needed.

Parameters

data : bytes

The extracted image bytes.

name : str

The image name given by the file.

Returns

DashAIImage

The image, with a .png name when it was converted.