TokenChunkModel
Chunking model that splits text into chunks based on token count.
Uses a HuggingFace AutoTokenizer to tokenize the text and splits at
token boundaries with configurable overlap.
Parameters
- tokenizer_name : string, default=
intfloat/e5-mistral-7b-instruct - The tokenizer model used to split the text into tokens.
- chunk_size : integer, default=
400 - Size of each chunk in tokens.
- chunk_overlap : integer, default=
40 - Number of tokens to overlap between chunks. Must be less than the chunk size.
Methods
chunk_text(self, text: str) -> List[str]
TokenChunkModelSplit text into chunks based on token count.
chunk(self, document: DashAI.back.dataloaders.classes.dashai_document.DashAIDocument) -> DashAI.back.dataloaders.classes.dashai_document.DashAIDocument
BaseChunkingModelSplit the text items of a document into a chunked document.
Parameters
- document : DashAIDocument
- The parsed source document.
Returns
- DashAIDocument
- One Text item of kind
"chunk"per chunk, withpageset to the first covered page andattrsholdingpages(first and last page) andsources(positions of the covered items).
get_credential(self, name: str)
ConfigObjectResolve a registered credential component by name.
Parameters
- name : str
- Credential component class name (e.g. "HuggingFaceCredential").
Returns
- BaseCredential
- An instance of the requested credential component.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.