Skip to main content

TokenChunkModel

ChunkingModel
DashAI.back.models.RAG.chunking_models.TokenChunkModel

Chunking model that splits text into chunks based on token count.

Uses a HuggingFace AutoTokenizer to tokenize the text and splits at token boundaries with configurable overlap.

Parameters​

tokenizer_name : string, default=intfloat/e5-mistral-7b-instruct
The tokenizer model to use for tokenizing the text.
chunk_size : integer, default=400
The size of each chunk in tokens.
chunk_overlap : integer, default=40
The number of overlapping tokens between chunks. Must be less than chunk_size.

Methods​

chunk_text(self, text: str) -> List[str]

Defined on TokenChunkModel

Split text into chunks based on token count.

chunk_document(self, document: DashAI.back.models.RAG.documents.base_document.BaseDocument) -> Dict[int, DashAI.back.models.RAG.documents.chunk.Chunk]

Defined on BaseChunkingModel

Chunk a single document using the chunk_text method.

chunk_documents(self) -> Dict[int, Dict[int, DashAI.back.models.RAG.documents.chunk.Chunk]]

Defined on BaseChunkingModel

Chunk documents using the chunk_document method.

compute_chunks(self) -> None

Defined on BaseChunkingModel

Compute chunks if not already done.

get_chunks(self) -> Dict[int, Dict[int, DashAI.back.models.RAG.documents.chunk.Chunk]]

Defined on BaseChunkingModel

Get the chunks generated by the chunking model.

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str
Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential
An instance of the requested credential component.

get_id(self) -> int

Defined on BaseChunkingModel

Get the ID of the chunking model.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict
Dictionary representing the Json Schema of the component.

set_id(self, id: int) -> None

Defined on BaseChunkingModel

Set the ID of the chunking model.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict
A dictionary with the data provided by the user to initialize the model.

Returns

dict
A validated dictionary with the necessary objects.