PdfPlumberDocumentDataLoader

DataLoader

DashAI.back.dataloaders.classes.PdfPlumberDocumentDataLoader

Parse a PDF with pdfplumber: per page, its text blocks, then its tables.

With 'block' granularity the lines of a page are grouped into blocks by the gap between them and by their font size, and a short block set larger than the body text is named a heading. With 'page' granularity the text of a page becomes one item.

Parameters

password, default=None: Password for encrypted PDFs. Leave empty for unencrypted files.
x_tolerance : number, default=3.0: Horizontal distance, in points, below which two characters belong to the same word. Lower it when words run together.
y_tolerance : number, default=3.0: Vertical distance, in points, below which two characters belong to the same line. Raise it when a line is split in two.
layout : boolean, default=False: Pad the output with whitespace so it mirrors the position of the text on the page. Preserves columns at the cost of extra whitespace.
extract_tables : boolean, default=False: Store each detected table as its own text item of tab separated rows, right after the text of its page.
granularity : string, default=block: How much text one item holds. One item per block groups the lines of a page into blocks and stores each one on its own; one item per page keeps the whole page together. A block never crosses a page boundary.
block_gap : number, default=0.6: Vertical gap between two lines, as a fraction of line height, above which a new block starts. Raise it to join lines that sit far apart. Only used when one item holds one block.
detect_headings : boolean, default=True: Name short blocks set in a font larger than the body text as headings, with a level from the rank of their size. Only used when one item holds one block.

Methods

load_document(self, file_path: str, params: Dict[str, Any]) -> 'DashAIDocument'

Defined on PdfPlumberDocumentDataLoader

Parse a PDF with pdfplumber.

Parameters

file_path : str: Path of the PDF file.
params : Dict[str, Any]: pdfplumber parameters, granularity, block_gap and detect_headings; missing keys take the defaults.

Returns

DashAIDocument: Text items for the blocks of each page, named as headings when they stand out, and table items with their rows, in reading order.

render_table(cls, table: List[List[object]]) -> str

Defined on PdfPlumberDocumentDataLoader

Render one table as tab separated rows.

Parameters

table : List[List[object]]: Rows of cells as pdfplumber returns them; empty cells are None.

Returns

str: Newline separated rows of tab separated cells.

table_rows(table: List[List[object]]) -> List[List[str]]

Defined on PdfPlumberDocumentDataLoader

Turn one pdfplumber table into rows of string cells.

Parameters

table : List[List[object]]: Rows of cells as pdfplumber returns them; empty cells are None.

Returns

List[List[str]]: The same rows with every cell as a string, empty cells as "".

build_metadata(self, file_path: str, params: Dict[str, Any], n_pages: Optional[int]) -> Dict[str, Any]

Defined on DocumentDataLoader

Build the metadata stored with the parsed document.

Parameters

file_path : str: Path of the parsed file.
params : Dict[str, Any]: The resolved loader parameters.
n_pages : Optional[int]: Number of pages, None when the format has no pages.

Returns

Dict[str, Any]: file_name, file_type, n_pages, dataloader, dataloader_params.

extract_files(self, file_path: str, temp_path: str) -> str

Defined on BaseDataLoader

Extract a ZIP archive into a subdirectory of temp_path.

Parameters

file_path : str: Path to the ZIP archive to extract.
temp_path : str: Base temporary directory; extraction target is <temp_path>/files/.

Returns

str: Path of the directory containing the extracted files (<temp_path>/files/).

extract_native_types(self, filepath_or_buffer: str, params: Dict[str, Any]) -> Optional[Dict[str, Dict[str, Any]]]

Defined on BaseDataLoader

Extract column types from the file's own metadata, if available.

Parameters

filepath_or_buffer : str: Path to the file already prepared on disk (post-ZIP extraction).
params : Dict[str, Any]: Dataloader parameters, same dict that load_preview receives.

Returns

Dict[str, Dict[str, Any]] | None: Column name -> DashAI type dict, or None if the format does not provide native types.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str: Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential: An instance of the requested credential component.

get_metadata(cls) -> Dict[str, Any]

Defined on BaseDataLoader

Return the registry metadata the frontend reads.

Returns

Dict[str, Any]: category, supported_extensions, tags (those extensions as upper case file formats), supports_native_types and data_kinds, the list of data kinds the loader produces.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict: Dictionary representing the Json Schema of the component.

load_data(self, filepath_or_buffer, temp_path, params, n_sample=None)

Defined on DocumentDataLoader

Refuse to build a dataset.

load_preview(self, filepath_or_buffer: str, params: Dict[str, Any], n_rows: int = 10) -> 'DataFrame'

Defined on BaseDataLoader

Load a preview of the dataset using streaming.

Parameters

filepath_or_buffer : str: Path to the file or buffer to load.
params : Dict[str, Any]: Parameters for loading the data.
n_rows : int, optional: Number of rows to preview. Default is 10.

Returns

DataFrame: A pandas DataFrame with the preview data.

prepare_files(self, file_path: str, temp_path: str) -> str

Defined on BaseDataLoader

Resolve a file path or URL into a local path suitable for loading.

Parameters

file_path : str: Path to a local file, a ZIP archive, or an HTTP(S) URL.
temp_path : str: Temporary directory used for extraction of ZIP or URL downloads.

Returns

tuple of (str, str): (path, type_path) where type_path is "dir" for extracted archives/URLs or "file" for plain local files.

resolve_params(self, params: Optional[Dict[str, Any]]) -> Dict[str, Any]

Defined on DocumentDataLoader

Fill the parameters the caller left out and validate them.

Parameters

params : Optional[Dict[str, Any]]: Loader parameters as the form sends them.

Returns

Dict[str, Any]: Every schema field with its validated value.

supports_file(cls, file_name: str) -> bool

Defined on DocumentDataLoader

Tell whether the loader accepts a file by its extension.

Parameters

file_name : str: Name or path of the file.

Returns

bool: True when the lowercased suffix is in SUPPORTED_EXTENSIONS.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict: A dictionary with the data provided by the user to initialize the model.

Returns

dict: A validated dictionary with the necessary objects.

web_image(data: bytes, name: str) -> 'DashAIImage'

Defined on DocumentDataLoader

Wrap extracted image bytes, converting them to PNG when needed.

Parameters

data : bytes: The extracted image bytes.
name : str: The image name given by the file.

Returns

DashAIImage: The image, with a .png name when it was converted.

Parameters​

Methods​

Parameters

Methods