Package chat.octet.model.beans

Class LlamaContextParams

java.lang.Object

chat.octet.model.beans.LlamaContextParams

public class LlamaContextParams
extends Object

Llama context params entity

Author:

William

Field Summary

Fields

Modifier and Type	Field	Description
int	batch	prompt processing batch size.
int	ctx	text context size.
int	dataTypeK	data type for K cache.
int	dataTypeV	data type for V cache.
float	defragThold	defragment the KV cache if holes/size > thold, invalid input: '<' 0 disabled (default).
boolean	embedding	embedding mode only.
boolean	flashAttn	whether to use flash attention [EXPERIMENTAL].
boolean	logitsAll	the llama_eval() call computes all logits, not just the last one.
boolean	offloadKqv	whether to offload the KQV ops (including the KV cache) to GPU.
int	poolingType	whether to pool (sum) embedding results by sequence id,(ignored if no pooling layer).
float	ropeFreqBase	RoPE base frequency.
float	ropeFreqScale	RoPE frequency scaling factor.
int	ropeScalingType	RoPE scaling type, from `enum llama_rope_scaling_type`.
int	seed	RNG seed, -1 for random.
int	seqMax	max number of sequences (i.e.
int	threads	number of threads used for generation.
int	threadsBatch	number of threads used for prompt and batch processing.
int	ubatch	physical maximum batch size.
float	yarnAttnFactor	YaRN magnitude scaling factor.
float	yarnBetaFast	YaRN low correction dim.
float	yarnBetaSlow	YaRN high correction dim.
float	yarnExtFactor	YaRN extrapolation mix factor, NaN = from model.
int	yarnOrigCtx	YaRN original context size.

Constructor Summary

Constructors

Constructor	Description
LlamaContextParams()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

seed

public int seed

RNG seed, -1 for random.

ctx

public int ctx

text context size.

batch

public int batch

prompt processing batch size.

ubatch

public int ubatch

physical maximum batch size.

seqMax

public int seqMax

max number of sequences (i.e. distinct states for recurrent models).

threads

public int threads

number of threads used for generation.

threadsBatch

public int threadsBatch

number of threads used for prompt and batch processing.

ropeScalingType

public int ropeScalingType

RoPE scaling type, from `enum llama_rope_scaling_type`.

See Also:

• LlamaRoPEScalingType

poolingType

public int poolingType

whether to pool (sum) embedding results by sequence id,(ignored if no pooling layer).

See Also:

• LlamaPoolingType

yarnExtFactor

public float yarnExtFactor

YaRN extrapolation mix factor, NaN = from model.

yarnAttnFactor

public float yarnAttnFactor

YaRN magnitude scaling factor.

yarnBetaFast

public float yarnBetaFast

YaRN low correction dim.

yarnBetaSlow

public float yarnBetaSlow

YaRN high correction dim.

yarnOrigCtx

public int yarnOrigCtx

YaRN original context size.

defragThold

public float defragThold

defragment the KV cache if holes/size > thold, invalid input: '<' 0 disabled (default).

ropeFreqBase

public float ropeFreqBase

RoPE base frequency.

ropeFreqScale

public float ropeFreqScale

RoPE frequency scaling factor.

dataTypeK

public int dataTypeK

data type for K cache.

dataTypeV

public int dataTypeV

data type for V cache.

logitsAll

public boolean logitsAll

the llama_eval() call computes all logits, not just the last one.

embedding

public boolean embedding

embedding mode only.

offloadKqv

public boolean offloadKqv

whether to offload the KQV ops (including the KV cache) to GPU.

flashAttn

public boolean flashAttn

whether to use flash attention [EXPERIMENTAL].

Constructor Details

LlamaContextParams

public LlamaContextParams()