PypdfDocumentDataLoader
Parse a PDF with pypdf: per page, its text blocks, then its images.
Parameters
- strict : boolean, default=
True - Fail on a malformed PDF instead of trying to recover from it.
- granularity : string, default=
block - How much text one item holds. One item per block splits a page wherever a blank line separates its text; one item per page keeps the whole page together. pypdf has no layout analysis, so a page whose text has no blank line stays a single item either way, while pdfminer.six finds real layout blocks.
- extract_images : boolean, default=
False - Also store the images embedded in each page. They are placed after the text of their page in reading order.
Methods
load_document(self, file_path: str, params: Dict[str, Any]) -> 'DashAIDocument'
PypdfDocumentDataLoaderParse a PDF with pypdf.
Parameters
- file_path : str
- Path of the PDF file.
- params : Dict[str, Any]
strict,granularityandextract_images; missing keys take the defaults.
Returns
- DashAIDocument
- Text and Image items in reading order.
build_metadata(self, file_path: str, params: Dict[str, Any], n_pages: Optional[int]) -> Dict[str, Any]
DocumentDataLoaderBuild the metadata stored with the parsed document.
Parameters
- file_path : str
- Path of the parsed file.
- params : Dict[str, Any]
- The resolved loader parameters.
- n_pages : Optional[int]
- Number of pages, None when the format has no pages.
Returns
- Dict[str, Any]
file_name,file_type,n_pages,dataloader,dataloader_params.
extract_files(self, file_path: str, temp_path: str) -> str
BaseDataLoaderExtract a ZIP archive into a subdirectory of temp_path.
Parameters
- file_path : str
- Path to the ZIP archive to extract.
- temp_path : str
- Base temporary directory; extraction target is
<temp_path>/files/.
Returns
- str
- Path of the directory containing the extracted files (
<temp_path>/files/).
extract_native_types(self, filepath_or_buffer: str, params: Dict[str, Any]) -> Optional[Dict[str, Dict[str, Any]]]
BaseDataLoaderExtract column types from the file's own metadata, if available.
Parameters
- filepath_or_buffer : str
- Path to the file already prepared on disk (post-ZIP extraction).
- params : Dict[str, Any]
- Dataloader parameters, same dict that
load_previewreceives.
Returns
- Dict[str, Dict[str, Any]] | None
- Column name -> DashAI type dict, or
Noneif the format does not provide native types.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_metadata(cls) -> Dict[str, Any]
BaseDataLoaderReturn the registry metadata the frontend reads.
Returns
- Dict[str, Any]
category,supported_extensions,tags(those extensions as upper case file formats),supports_native_typesanddata_kinds, the list of data kinds the loader produces.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
load_data(self, filepath_or_buffer, temp_path, params, n_sample=None)
DocumentDataLoaderRefuse to build a dataset.
load_preview(self, filepath_or_buffer: str, params: Dict[str, Any], n_rows: int = 10) -> 'DataFrame'
BaseDataLoaderLoad a preview of the dataset using streaming.
Parameters
- filepath_or_buffer : str
- Path to the file or buffer to load.
- params : Dict[str, Any]
- Parameters for loading the data.
- n_rows : int, optional
- Number of rows to preview. Default is 10.
Returns
- DataFrame
- A pandas DataFrame with the preview data.
prepare_files(self, file_path: str, temp_path: str) -> str
BaseDataLoaderResolve a file path or URL into a local path suitable for loading.
Parameters
- file_path : str
- Path to a local file, a ZIP archive, or an HTTP(S) URL.
- temp_path : str
- Temporary directory used for extraction of ZIP or URL downloads.
Returns
- tuple of (str, str)
(path, type_path)wheretype_pathis"dir"for extracted archives/URLs or"file"for plain local files.
resolve_params(self, params: Optional[Dict[str, Any]]) -> Dict[str, Any]
DocumentDataLoaderFill the parameters the caller left out and validate them.
Parameters
- params : Optional[Dict[str, Any]]
- Loader parameters as the form sends them.
Returns
- Dict[str, Any]
- Every schema field with its validated value.
supports_file(cls, file_name: str) -> bool
DocumentDataLoaderTell whether the loader accepts a file by its extension.
Parameters
- file_name : str
- Name or path of the file.
Returns
- bool
- True when the lowercased suffix is in
SUPPORTED_EXTENSIONS.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.
web_image(data: bytes, name: str) -> 'DashAIImage'
DocumentDataLoaderWrap extracted image bytes, converting them to PNG when needed.
Parameters
- data : bytes
- The extracted image bytes.
- name : str
- The image name given by the file.
Returns
- DashAIImage
- The image, with a
.pngname when it was converted.