Skip to main content

Qwen3_8B

GenerativeModel
DashAI.back.models.hugging_face.Qwen3_8B

Qwen3 8B GGUF checkpoint (Q6_K quantization).

An 8B-parameter Qwen3 model distributed by Qwen in GGUF format. The Q6_K checkpoint offers higher weight precision than smaller quantizations, at the cost of a larger download and runtime memory requirement.

The checkpoint exposes an enable_thinking schema parameter, off by default because DashAI uses it as the RAG function-calling model. When off, the Qwen3 chat template appends an empty think block, so the model answers or emits its tool call without a reasoning trace. Turning it on restores the template's normal reasoning output for any chat use of this checkpoint.

References​

Parameters​

max_tokens : integer, default=100
Maximum number of new tokens the model will generate per response. Roughly 1 token ≈ 0.75 English words. Set to 100-200 for short answers, 500-1000 for detailed explanations or code. Must not exceed the context window minus the prompt length.
temperature : number, default=0.7
Sampling temperature controlling output randomness (range 0.0-1.0). At 0.0 the model always picks the most likely token (greedy, fully deterministic). Around 0.7 is a good balance for conversational tasks. At 1.0 outputs are maximally varied and unpredictable.
frequency_penalty : number, default=0.1
Penalizes tokens that have already appeared in the output based on how often they occur (range 0.0-2.0). At 0.0 there is no penalty and the model may repeat itself. Values around 0.1-0.3 gently discourage repetition. High values (1.5+) strongly prevent reuse of any word, which may produce less coherent text.
context_window : integer, default=512
Total token budget for a single forward pass, including both the input prompt and the generated response. Larger values allow longer conversations but consume more RAM/VRAM. Llama 3.1 supports up to 128K tokens natively; Llama 3.2 models support up to 128K tokens.
device : string, default=CPU
Hardware device for llama.cpp inference. 'CPU' runs the model fully in RAM with no GPU requirement. Selecting a GPU option offloads all layers for faster inference, setting n_gpu_layers=-1 so every transformer layer is GPU-accelerated.
enable_thinking : boolean, default=False
Emit a reasoning trace before the answer. It is off by default so RAG function calls stay within their token budget.

Methods​

close(self) -> None

Defined on GGUFTextGenerationModel

Release the llama.cpp backend and reclaim its accelerator memory.

Returns

None
Nothing.

component_dir(cls) -> pathlib.Path

Defined on DownloadableMixin

Return this component's own storage directory.

Returns

pathlib.Path
<COMPONENT_PATH>/<ClassName>.

delete(cls) -> None

Defined on DownloadableMixin

Remove the component's downloaded artifacts.

download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None

Defined on HFDownloadableMixin

Download all repos listed in hf_repos() into component_dir().

Parameters

report : ProgressReporter, optional
Callback invoked before each repo download with report(None, "Downloading <repo_id>"). None means no progress reporting.

generate(self, prompt: List[Dict[str, str]]) -> List[str]

Defined on GGUFTextGenerationModel

Generate a reply for the given chat prompt.

Parameters

prompt : list of dict
Conversation history in OpenAI chat format. Each dict must contain at least "role" ("system", "user", or "assistant") and "content" (the message text).

Returns

list of str
A single-element list containing the model's reply text, extracted from choices[0]["message"]["content"].

generate_function_call(self, messages: List[Dict[str, str]], functions: List[Dict[str, Any]]) -> DashAI.back.models.text_to_text_generation_model.FunctionCallResult

Defined on GGUFTextGenerationModel

Produce a named function call via constrained sampling.

Parameters

messages : List[Dict[str, str]]
Chat messages (OpenAI format) as passed to generate.
functions : List[Dict[str, Any]]
OpenAI-style function specs. The first spec is the one the model is forced to call.

Returns

FunctionCallResult
The called function name and its raw arguments payload (the JSON-encoded string emitted by llama.cpp, left unresolved for the caller).

get_credential(self, name: str)

Defined on ConfigObject

Resolve a registered credential component by name.

Parameters

name : str
Credential component class name (e.g. "HuggingFaceCredential").

Returns

BaseCredential
An instance of the requested credential component.

get_metadata(cls) -> Dict[str, Any]

Defined on BaseGenerativeModel

Get metadata values for the current generative model.

Returns

Dict[str, Any]
Dictionary indicating whether the model requires a download before use and the expected download size in bytes.

get_schema(cls) -> dict

Defined on ConfigObject

Generates the component related Json Schema.

Returns

dict
Dictionary representing the Json Schema of the component.

hf_repos(cls) -> List[Tuple[str, str, List[str]]]

Defined on GGUFTextGenerationModel

Return the single HuggingFace repo entry for this checkpoint.

Returns

list of tuple
A list containing one 3-tuple (repo_id, "model", [gguf_pattern]) when REPO_ID is set, or an empty list otherwise.

is_downloaded(cls) -> bool

Defined on GGUFTextGenerationModel

Return whether a GGUF file matching the checkpoint pattern exists.

Returns

bool
True when every repo in hf_repos() contains at least one file matching GGUF_PATTERN; False otherwise.

supports_function_calling(cls) -> bool

Defined on TextToTextGenerationTaskModel

Report whether instances of this class support function calling.

Returns

bool
True when the concrete backend implements generate_function_call with schema-constrained sampling.

validate_and_transform(self, raw_data: dict) -> dict

Defined on ConfigObject

It takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.

Parameters

raw_data : dict
A dictionary with the data provided by the user to initialize the model.

Returns

dict
A validated dictionary with the necessary objects.

Compatible with​