Llama31_8BInstruct
Meta Llama 3.1 8B Instruct GGUF checkpoint (Q4_K_M quantization).
An 8B-parameter instruction-tuned model from Meta with strong general reasoning and multilingual ability. Weights are stored locally after a one-time download from HuggingFace.
References
Parameters
- max_tokens : integer, default=
100 - Maximum number of new tokens the model will generate per response. Roughly 1 token ≈ 0.75 English words. Set to 100-200 for short answers, 500-1000 for detailed explanations or code. Must not exceed the context window minus the prompt length.
- temperature : number, default=
0.7 - Sampling temperature controlling output randomness (range 0.0-1.0). At 0.0 the model always picks the most likely token (greedy, fully deterministic). Around 0.7 is a good balance for conversational tasks. At 1.0 outputs are maximally varied and unpredictable.
- frequency_penalty : number, default=
0.1 - Penalizes tokens that have already appeared in the output based on how often they occur (range 0.0-2.0). At 0.0 there is no penalty and the model may repeat itself. Values around 0.1-0.3 gently discourage repetition. High values (1.5+) strongly prevent reuse of any word, which may produce less coherent text.
- context_window : integer, default=
512 - Total token budget for a single forward pass, including both the input prompt and the generated response. Larger values allow longer conversations but consume more RAM/VRAM. Llama 3.1 supports up to 128K tokens natively; Llama 3.2 models support up to 128K tokens.
- device : string, default=
CPU - Hardware device for llama.cpp inference. 'CPU' runs the model fully in RAM with no GPU requirement. Selecting a GPU option offloads all layers for faster inference, setting n_gpu_layers=-1 so every transformer layer is GPU-accelerated.
Methods
component_dir(cls) -> pathlib.Path
DownloadableMixinReturn this component's own storage directory.
Returns
- pathlib.Path
<COMPONENT_PATH>/<ClassName>.
delete(cls) -> None
DownloadableMixinRemove the component's downloaded artifacts.
download(cls, report: Optional[Callable[[Optional[float], Optional[str]], NoneType]] = None) -> None
HFDownloadableMixinDownload all repos listed in hf_repos() into component_dir().
Parameters
- report : ProgressReporter, optional
- Callback invoked before each repo download with
report(None, "Downloading <repo_id>").Nonemeans no progress reporting.
generate(self, prompt: list) -> List[str]
GGUFTextGenerationModelGenerate a reply for the given chat prompt.
Parameters
- prompt : list of dict
- Conversation history in OpenAI chat format. Each dict must contain at least
"role"("system","user", or"assistant") and"content"(the message text).
Returns
- list of str
- A single-element list containing the model's reply text, extracted from
choices[0]["message"]["content"].
get_metadata(cls) -> Dict[str, Any]
BaseGenerativeModelGet metadata values for the current generative model.
Returns
- Dict[str, Any]
- Dictionary indicating whether the model requires a download before use and the expected download size in bytes.
get_schema(cls) -> dict
ConfigObjectGenerates the component related Json Schema.
Returns
- dict
- Dictionary representing the Json Schema of the component.
hf_repos(cls) -> List[tuple]
GGUFTextGenerationModelReturn the single HuggingFace repo entry for this checkpoint.
Returns
- list of tuple
- A list containing one 3-tuple
(repo_id, "model", [gguf_pattern])whenREPO_IDis set, or an empty list otherwise.
is_downloaded(cls) -> bool
HFDownloadableMixinReturn whether all repo directories exist and are non-empty.
Returns
- bool
Truewhen every repo listed inhf_repos()has a non-empty local directory;Falseotherwise (including when the list is empty).
validate_and_transform(self, raw_data: dict) -> dict
ConfigObjectIt takes the data given by the user to initialize the model and returns it with all the objects that the model needs to work.
Parameters
- raw_data : dict
- A dictionary with the data provided by the user to initialize the model.
Returns
- dict
- A validated dictionary with the necessary objects.