Package chat.octet.model.beans

Class LlamaModelParams

java.lang.Object

chat.octet.model.beans.LlamaModelParams

public class LlamaModelParams
extends Object

Llama model params entity

Author:

William

Field Summary

Fields

Modifier and Type	Field	Description
boolean	checkTensors	validate model tensor data.
int	gpuLayers	number of layers to store in VRAM.
int	mainGpu	the GPU that is used for scratch and small tensors.
boolean	mlock	force system to keep model in RAM.
boolean	mmap	use mmap if possible.
int	splitMode	how to split the model across multiple GPUs.
float[]	tensorSplit	how to split layers across multiple GPUs (size: LLAMA_MAX_DEVICES).
boolean	vocabOnly	only load the vocabulary, no weights.

Constructor Summary

Constructors

Constructor	Description
LlamaModelParams()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

gpuLayers

public int gpuLayers

number of layers to store in VRAM.

splitMode

public int splitMode

how to split the model across multiple GPUs.

LLAMA_SPLIT_NONE = 0 (single GPU) LLAMA_SPLIT_LAYER = 1 (split layers and KV across GPUs) LLAMA_SPLIT_ROW = 2 (split rows across GPUs)

mainGpu

public int mainGpu

the GPU that is used for scratch and small tensors.

tensorSplit

public float[] tensorSplit

how to split layers across multiple GPUs (size: LLAMA_MAX_DEVICES).

vocabOnly

public boolean vocabOnly

only load the vocabulary, no weights.

mmap

public boolean mmap

use mmap if possible.

mlock

public boolean mlock

force system to keep model in RAM.

checkTensors

public boolean checkTensors

validate model tensor data.

Constructor Details

LlamaModelParams

public LlamaModelParams()