PdfMinerDocumentDataLoader
Parse a PDF with pdfminer.six into Text items.
With 'block' granularity every layout block pdfminer finds becomes its own item, and a block set in a larger font than the body text is named a heading. With 'page' granularity the text of a page becomes one item: pdfminer's text converter writes a form feed after every page, so the document text split on it gives the pages in order.
Parameters
- password, default=
None - Password for encrypted PDFs. Leave empty for unencrypted files.
- line_margin : number, default=
0.5 - Vertical gap, as a fraction of line height, below which two lines belong to the same paragraph. Raise it to merge tight line spacing.
- char_margin : number, default=
2.0 - Horizontal gap, as a fraction of character width, below which two characters belong to the same word. Lower it to split joined text.
- boxes_flow : number, default=
0.5 - How much reading order follows horizontal position over vertical position. -1.0 is strictly left to right, 1.0 strictly top to bottom.
- detect_vertical : boolean, default=
False - Detect vertically written text, as used in CJK layouts.
- granularity : string, default=
block - How much text one item holds. One item per block stores every layout block pdfminer finds, such as a paragraph or a heading; one item per page keeps the whole page together. A block never crosses a page boundary.
- detect_headings : boolean, default=
True - Name short blocks set in a font larger than the body text as headings, with a level from the rank of their size. Only used when one item holds one block.
Methods
load_document(self, file_path: str, params: Dict[str, Any]) -> 'DashAIDocument'
PdfMinerDocumentDataLoaderParse a PDF with pdfminer.six.
Parameters
- file_path : str
- Path of the PDF file.
- params : Dict[str, Any]
- Layout parameters,
granularityanddetect_headings; missing keys take the defaults.
Returns
- DashAIDocument
- One Text item per layout block, or one per page with text, in reading order. Short blocks set larger than the body text are named headings when
detect_headingsis on.
build_metadata(self, file_path: str, params: Dict[str, Any], n_pages: Optional[int]) -> Dict[str, Any]
DocumentDataLoaderBuild the metadata stored with the parsed document.
Parameters
- file_path : str
- Path of the parsed file.
- params : Dict[str, Any]
- The resolved loader parameters.
- n_pages : Optional[int]
- Number of pages, None when the format has no pages.
Returns
- Dict[str, Any]
file_name,file_type,n_pages,dataloader,dataloader_params.
extract_files(self, file_path: str, temp_path: str) -> str
BaseDataLoaderExtract a ZIP archive into a subdirectory of temp_path.
Parameters
- file_path : str
- Path to the ZIP archive to extract.
- temp_path : str
- Base temporary directory; extraction target is
<temp_path>/files/.
Returns
- str
- Path of the directory containing the extracted files (
<temp_path>/files/).
extract_native_types(self, filepath_or_buffer: str, params: Dict[str, Any]) -> Optional[Dict[str, Dict[str, Any]]]
BaseDataLoaderExtract column types from the file's own metadata, if available.
Parameters
- filepath_or_buffer : str
- Path to the file already prepared on disk (post-ZIP extraction).
- params : Dict[str, Any]
- Dataloader parameters, same dict that
load_previewreceives.
Returns
- Dict[str, Dict[str, Any]] | None
- Column name -> DashAI type dict, or
Noneif the format does not provide native types.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_metadata(cls) -> Dict[str, Any]
BaseDataLoaderReturn the registry metadata the frontend reads.
Returns
- Dict[str, Any]
category,supported_extensions,tags(those extensions as upper case file formats),supports_native_typesanddata_kinds, the list of data kinds the loader produces.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
load_data(self, filepath_or_buffer, temp_path, params, n_sample=None)
DocumentDataLoaderRefuse to build a dataset.
load_preview(self, filepath_or_buffer: str, params: Dict[str, Any], n_rows: int = 10) -> 'DataFrame'
BaseDataLoaderLoad a preview of the dataset using streaming.
Parameters
- filepath_or_buffer : str
- Path to the file or buffer to load.
- params : Dict[str, Any]
- Parameters for loading the data.
- n_rows : int, optional
- Number of rows to preview. Default is 10.
Returns
- DataFrame
- A pandas DataFrame with the preview data.
prepare_files(self, file_path: str, temp_path: str) -> str
BaseDataLoaderResolve a file path or URL into a local path suitable for loading.
Parameters
- file_path : str
- Path to a local file, a ZIP archive, or an HTTP(S) URL.
- temp_path : str
- Temporary directory used for extraction of ZIP or URL downloads.
Returns
- tuple of (str, str)
(path, type_path)wheretype_pathis"dir"for extracted archives/URLs or"file"for plain local files.
resolve_params(self, params: Optional[Dict[str, Any]]) -> Dict[str, Any]
DocumentDataLoaderFill the parameters the caller left out and validate them.
Parameters
- params : Optional[Dict[str, Any]]
- Loader parameters as the form sends them.
Returns
- Dict[str, Any]
- Every schema field with its validated value.
supports_file(cls, file_name: str) -> bool
DocumentDataLoaderTell whether the loader accepts a file by its extension.
Parameters
- file_name : str
- Name or path of the file.
Returns
- bool
- True when the lowercased suffix is in
SUPPORTED_EXTENSIONS.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.
web_image(data: bytes, name: str) -> 'DashAIImage'
DocumentDataLoaderWrap extracted image bytes, converting them to PNG when needed.
Parameters
- data : bytes
- The extracted image bytes.
- name : str
- The image name given by the file.
Returns
- DashAIImage
- The image, with a
.pngname when it was converted.