UnstructuredDocumentDataLoader
Parse office, web and email files with unstructured's partitioners.
Each element unstructured returns becomes one item named after its category: Title becomes a heading, ListItem a list item, Table a table with its rows, Header and Footer page headers and footers, FigureCaption a caption and CodeSnippet code; every other text element is a paragraph. Heading levels and list depths count from the shallowest one in the file. unstructured's usage analytics and notice are turned off.
unstructured classifies text with the spaCy model en_core_web_sm,
which it would otherwise download and install into the environment on
first use. Here it is a component download instead: the pinned wheel is
unpacked into component_dir(), which is put on sys.path while
parsing, and the loader refuses to parse until it is present.
Parameters
- include_headers_footers : boolean, default=
False - Keep the page headers and footers unstructured detects.
Methods
download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None
UnstructuredDocumentDataLoaderDownload the pinned spaCy model wheel and unpack it.
Parameters
- report : ProgressReporter, optional
- Called with the fraction downloaded each time it grows by a percent.
is_downloaded(cls) -> bool
UnstructuredDocumentDataLoaderReturn whether the spaCy model is unpacked in component_dir().
Returns
- bool
- True when both the model package and its
dist-infofolder, whichspacy.loadneeds to find it by name, are present.
load_document(self, file_path: str, params: Dict[str, Any]) -> 'DashAIDocument'
UnstructuredDocumentDataLoaderParse a file with unstructured.
Parameters
- file_path : str
- Path of the file.
- params : Dict[str, Any]
include_headers_footers; missing keys take the default.
Returns
- DashAIDocument
- One text item per element in reading order, with the slide or sheet number as page where unstructured gives one.
table_rows(html: str) -> Tuple[List[List[str]], bool]
UnstructuredDocumentDataLoaderRead the rows of a table from the HTML unstructured gives it.
Parameters
- html : str
- The
text_as_htmlmetadata of a Table element.
Returns
- Tuple[List[List[str]], bool]
- Each row as its cells' text, and whether the first row is made of header cells only.
build_metadata(self, file_path: str, params: Dict[str, Any], n_pages: Optional[int]) -> Dict[str, Any]
DocumentDataLoaderBuild the metadata stored with the parsed document.
Parameters
- file_path : str
- Path of the parsed file.
- params : Dict[str, Any]
- The resolved loader parameters.
- n_pages : Optional[int]
- Number of pages, None when the format has no pages.
Returns
- Dict[str, Any]
file_name,file_type,n_pages,dataloader,dataloader_params.
component_dir(cls) -> pathlib.Path
DownloadableMixinReturn this component's own storage directory.
Returns
- pathlib.Path
<COMPONENT_PATH>/<ClassName>.
delete(cls) -> None
DownloadableMixinRemove the component's downloaded artifacts.
extract_files(self, file_path: str, temp_path: str) -> str
BaseDataLoaderExtract a ZIP archive into a subdirectory of temp_path.
Parameters
- file_path : str
- Path to the ZIP archive to extract.
- temp_path : str
- Base temporary directory; extraction target is
<temp_path>/files/.
Returns
- str
- Path of the directory containing the extracted files (
<temp_path>/files/).
extract_native_types(self, filepath_or_buffer: str, params: Dict[str, Any]) -> Optional[Dict[str, Dict[str, Any]]]
BaseDataLoaderExtract column types from the file's own metadata, if available.
Parameters
- filepath_or_buffer : str
- Path to the file already prepared on disk (post-ZIP extraction).
- params : Dict[str, Any]
- Dataloader parameters, same dict that
load_previewreceives.
Returns
- Dict[str, Dict[str, Any]] | None
- Column name -> DashAI type dict, or
Noneif the format does not provide native types.
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_metadata(cls) -> Dict[str, Any]
BaseDataLoaderReturn the registry metadata the frontend reads.
Returns
- Dict[str, Any]
category,supported_extensions,tags(those extensions as upper case file formats),supports_native_types,data_kinds, the list of data kinds the loader produces, andrequires_downloadwithdownload_size_bytesfor loaders that must download artifacts before use.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
load_data(self, filepath_or_buffer, temp_path, params, n_sample=None)
DocumentDataLoaderRefuse to build a dataset.
load_preview(self, filepath_or_buffer: str, params: Dict[str, Any], n_rows: int = 10) -> 'DataFrame'
BaseDataLoaderLoad a preview of the dataset using streaming.
Parameters
- filepath_or_buffer : str
- Path to the file or buffer to load.
- params : Dict[str, Any]
- Parameters for loading the data.
- n_rows : int, optional
- Number of rows to preview. Default is 10.
Returns
- DataFrame
- A pandas DataFrame with the preview data.
prepare_files(self, file_path: str, temp_path: str) -> str
BaseDataLoaderResolve a file path or URL into a local path suitable for loading.
Parameters
- file_path : str
- Path to a local file, a ZIP archive, or an HTTP(S) URL.
- temp_path : str
- Temporary directory used for extraction of ZIP or URL downloads.
Returns
- tuple of (str, str)
(path, type_path)wheretype_pathis"dir"for extracted archives/URLs or"file"for plain local files.
resolve_params(self, params: Optional[Dict[str, Any]]) -> Dict[str, Any]
DocumentDataLoaderFill the parameters the caller left out and validate them.
Parameters
- params : Optional[Dict[str, Any]]
- Loader parameters as the form sends them.
Returns
- Dict[str, Any]
- Every schema field with its validated value.
supports_file(cls, file_name: str) -> bool
DocumentDataLoaderTell whether the loader accepts a file by its extension.
Parameters
- file_name : str
- Name or path of the file.
Returns
- bool
- True when the lowercased suffix is in
SUPPORTED_EXTENSIONS.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.
web_image(data: bytes, name: str) -> 'DashAIImage'
DocumentDataLoaderWrap extracted image bytes, converting them to PNG when needed.
Parameters
- data : bytes
- The extracted image bytes.
- name : str
- The image name given by the file.
Returns
- DashAIImage
- The image, with a
.pngname when it was converted.