CharacterChunkModel

ChunkingModel

DashAI.back.models.RAG.chunking_models.CharacterChunkModel

---

Chunking model that splits documents into chunks based on character count. This model is useful for processing large text documents into manageable pieces.

Parameters

chunk_size : integer, default=500

Size of each chunk in characters.

chunk_overlap : integer, default=50

Number of characters to overlap between chunks. Must be less than the chunk size.

Methods

chunk_text(self, text: str) -> List[str]

Defined on CharacterChunkModel

Chunk the input text into smaller segments based on character count.

chunk(self, document: DashAI.back.dataloaders.classes.dashai_document.DashAIDocument) -> DashAI.back.dataloaders.classes.dashai_document.DashAIDocument

Defined on BaseChunkingModel

Split the text items of a document into a chunked document.

Parameters

document : DashAIDocument

The parsed source document.

Returns

DashAIDocument

One Text item of kind "chunk" per chunk, with page set to the first covered page and attrs holding pages (first and last page) and sources (positions of the covered items).

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str

Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential

An instance of the requested credential component.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict

Dictionary representing the Json Schema of the component.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict

A dictionary with the data provided by the user to initialize the model.

Returns

dict

A validated dictionary with the necessary objects.