Model configuration for a single tier. More...
#include <entropic/types/config.h>
Public Attributes | |
| std::filesystem::path | path |
| Resolved model file path. | |
| std::string | adapter = "qwen35" |
| Chat adapter name. | |
| int | context_length = 16384 |
| Context window size (512–131072) | |
| int | gpu_layers = -1 |
| GPU offload layers (-1 = all) | |
| bool | gpu_layers_auto = false |
Derive gpu_layers from free VRAM at admission (gh#148). | |
| bool | keep_warm = false |
| Keep this model resident in host RAM (WARM) when it leaves the active slot, instead of unloading it (gh#157). | |
| bool | use_mlock = true |
| Lock model in system RAM. | |
| int | cpu_moe_layers = 0 |
EXPERIMENTAL: layers whose ROUTED-EXPERT tensors go to the host while everything else follows gpu_layers (gh#153, decision #42(iii)). | |
| int | reasoning_budget = -1 |
| Think token budget (-1 = unlimited) | |
| std::string | cache_type_k = "f16" |
| KV cache key quantization type. | |
| std::string | cache_type_v = "f16" |
| KV cache value quantization type. | |
| int | n_batch = 512 |
| Batch size for prompt processing. | |
| int | n_ubatch = 0 |
| Physical micro-batch size for prompt processing (gh#23 MVP item 5). | |
| int | n_threads = 0 |
| CPU threads (0 = auto-detect) | |
| std::string | tensor_split |
| Multi-GPU tensor split ratios (empty = single GPU) | |
| std::string | split_mode |
| Multi-GPU split mode for model load (gh#23 MVP item 6). | |
| int | main_gpu = 0 |
| Primary GPU index for model load (gh#23 MVP item 7). | |
| bool | offload_kqv = true |
| Offload KQV ops (incl. | |
| float | rope_freq_base = 0.0f |
| RoPE base frequency override (gh#23 MVP item 9). | |
| float | rope_freq_scale = 0.0f |
| RoPE frequency scaling factor (gh#23 MVP item 10). | |
| int | n_parallel = 1 |
| Max parallel sequences per context (gh#23 MVP item 11). | |
| int | max_sessions = 1 |
| Resident per-caller conversation sessions this tier keeps KV for (gh#144). | |
| bool | flash_attn = true |
| Enable flash attention. | |
| std::optional< std::vector< std::string > > | allowed_tools |
| Tool whitelist (nullopt = all) | |
| std::filesystem::path | mmproj_path |
| Vision projector GGUF path. | |
| std::string | model_format = "gguf" |
| Expected model format. | |
Detailed Description
Model configuration for a single tier.
Contains all parameters needed to load and configure a model, including llama.cpp pass-through fields for KV cache, batching, threading, and attention.
Every scalar field carries its default as a member initializer here — this struct IS the default spec, not the YAML loader. std::optional fields (e.g. allowed_tools) mean "absent", resolved elsewhere, and are never collapsed into a concrete default.
@req REQ-TYPE-005
- Version
- 1.8.0
Member Data Documentation
◆ adapter
| std::string entropic::ModelConfig::adapter = "qwen35" |
◆ allowed_tools
| std::optional<std::vector<std::string> > entropic::ModelConfig::allowed_tools |
◆ cache_type_k
| std::string entropic::ModelConfig::cache_type_k = "f16" |
◆ cache_type_v
| std::string entropic::ModelConfig::cache_type_v = "f16" |
◆ context_length
| int entropic::ModelConfig::context_length = 16384 |
◆ cpu_moe_layers
| int entropic::ModelConfig::cpu_moe_layers = 0 |
EXPERIMENTAL: layers whose ROUTED-EXPERT tensors go to the host while everything else follows gpu_layers (gh#153, decision #42(iii)).
gpu_layers is role-blind — a layer is wholly on the card or wholly on the host — so a model that does not fit spends VRAM on rarely touched expert weights and exiles attention AND ITS KV to system RAM. This places the first N layers' expert FFN tensors (ffn_{gate,up,down,gate_up}_exps and their per-expert scales) on the CPU; attention, KV, the router, the shared/dense FFN, norms and embeddings are untouched and still follow gpu_layers.
It maps 1:1 onto llama.cpp's --n-cpu-moe / -ncmoe (LLAMA_ARG_N_CPU_MOE) and is named for it deliberately: the mechanism, the semantics ("the FIRST N layers") and the tensor patterns are upstream's, so an operator who has read llama.cpp's docs transfers what they know without translation.
0 (default) installs no overrides at all and is byte-identical to every release before v2.13.0. There is no "all layers" sentinel: negative values are REFUSED, not reinterpreted.
- Warning
- PROTOTYPE, pending measurement. The gh#148 residency math (
estimate_footprint_bytes,gpu_layers: auto) prices a layer as a whole layer and knows nothing about expert placement, sogpu_layers: autois refused in combination with this key and an explicitgpu_layersis required. Also refused: a negative count,gpu_layers: 0(nothing is on the card to move off), a model that declares no experts, and a count beyond the model's layer count — the last two at load time, since they need GGUF metadata.
- Version
- 2.13.0
◆ flash_attn
◆ gpu_layers
| int entropic::ModelConfig::gpu_layers = -1 |
◆ gpu_layers_auto
Derive gpu_layers from free VRAM at admission (gh#148).
Set by gpu_layers: auto in YAML. The engine then computes the split from the GGUF's size and the VRAM the device reports free, and LOGS the number it chose. Opt-in, because a derived split is a decision the operator should have asked for: the engine otherwise takes gpu_layers verbatim, and a clamp it applied on its own would be exactly the silent behaviour this codebase refuses.
It derives the LAYER SPLIT and nothing else — in particular it does not touch use_mlock, which the model-test harness used to flip off behind the operator's back for oversized models. That combination is refused loudly instead.
- Version
- 2.13.0
◆ keep_warm
Keep this model resident in host RAM (WARM) when it leaves the active slot, instead of unloading it (gh#157).
Read by the swap-out path only. It has never controlled STARTUP loading, though it was documented as "pre-warm model at startup" from v1.8.0 to v2.12.2 — the default tier loaded at init regardless of this flag. Startup behaviour is ModelsConfig::defer_load.
- Version
- 2.13.0
◆ main_gpu
| int entropic::ModelConfig::main_gpu = 0 |
Primary GPU index for model load (gh#23 MVP item 7).
llama.cpp's mparams.main_gpu. Effective when split_mode == "none" (single-GPU pinning) or "row" (small tensors go to this GPU). Ignored when split_mode == "layer". 0 (default) preserves pre-v2.3.19 behavior bit-for-bit.
- Version
- 2.3.19
◆ max_sessions
| int entropic::ModelConfig::max_sessions = 1 |
Resident per-caller conversation sessions this tier keeps KV for (gh#144).
1 (default) is bit-identical to pre-v2.12.0. Greater than 1 DERIVES the context geometry rather than being set alongside it — n_seq_max, kv_unified and n_ctx are computed from this and context_length, and the derivation is logged. Three independently settable knobs an operator can get inconsistent is precisely the silent-misconfiguration shape this codebase's fail-fast rule exists to prevent, and kv_unified is not a config key at all.
context_length stays PER SESSION: a pool of 3 at 32768 allocates 98304 cells, not 3 sessions sharing 32768.
- Version
- 2.12.0
◆ mmproj_path
| std::filesystem::path entropic::ModelConfig::mmproj_path |
◆ model_format
| std::string entropic::ModelConfig::model_format = "gguf" |
Expected model format.
"gguf" (default), "axmodel", "onnx", or empty (auto-detect). The backend validates that the file matches the expected format during load(). Mismatch returns ENTROPIC_ERROR_LOAD_FAILED with a diagnostic message identifying the actual format.
- Version
- 1.9.13
◆ n_batch
| int entropic::ModelConfig::n_batch = 512 |
◆ n_parallel
| int entropic::ModelConfig::n_parallel = 1 |
Max parallel sequences per context (gh#23 MVP item 11).
llama.cpp's cparams.n_seq_max. 1 (default) matches llama.cpp's default — single-sequence context, bit-identical pre-v2.3.23 behavior. Raising this enables KV-cache slot reuse across multiple concurrent generations (e.g. speculative rejection batches, batched-server scenarios). Effective max is LLAMA_MAX_SEQ; consult llama.cpp for the current ceiling.
- Version
- 2.3.23
◆ n_threads
| int entropic::ModelConfig::n_threads = 0 |
◆ n_ubatch
| int entropic::ModelConfig::n_ubatch = 0 |
Physical micro-batch size for prompt processing (gh#23 MVP item 5).
llama.cpp's cparams.n_ubatch. Decoupled from n_batch since llama.cpp v0.4 — n_batch is the LOGICAL batch (max tokens queued per llama_decode call) and n_ubatch is the PHYSICAL chunk the kernels actually process. Smaller n_ubatch reduces peak GPU memory for the same n_batch. 0 (default) means "match `n_batch`" — preserves pre-v2.3.17 behavior bit-for-bit since llama.cpp's default in that case is min(n_batch, default). Typical productive values: 128, 256, 512 (== n_batch).
- Version
- 2.3.17
◆ offload_kqv
Offload KQV ops (incl.
KV cache) to the GPU (gh#23 MVP item 8). llama.cpp's cparams.offload_kqv. true (default) matches llama.cpp's default — KQV runs on GPU for max throughput. Set false to keep KQV on the CPU side; saves VRAM at a throughput cost. Useful for tight-VRAM single-GPU setups.
- Version
- 2.3.20
◆ path
| std::filesystem::path entropic::ModelConfig::path |
◆ reasoning_budget
| int entropic::ModelConfig::reasoning_budget = -1 |
◆ rope_freq_base
| float entropic::ModelConfig::rope_freq_base = 0.0f |
RoPE base frequency override (gh#23 MVP item 9).
llama.cpp's cparams.rope_freq_base. 0.0 (default) takes the model's trained value — preserves pre-v2.3.21 behavior bit-for-bit. Positive overrides typically range 10000–10000000; raising it stretches the RoPE period (extends effective context at a quality cost). Pair with rope_freq_scale for YaRN-style context-extension setups.
- Version
- 2.3.21
◆ rope_freq_scale
| float entropic::ModelConfig::rope_freq_scale = 0.0f |
RoPE frequency scaling factor (gh#23 MVP item 10).
llama.cpp's cparams.rope_freq_scale. 0.0 (default) takes the model's trained value — preserves pre-v2.3.22 behavior bit-for-bit. Values in (0, 1) shrink the effective context (denser RoPE positions); values > 1 stretch it. Typical YaRN-style extension uses values like 0.5 (2× context). Pairs with rope_freq_base.
- Version
- 2.3.22
◆ split_mode
| std::string entropic::ModelConfig::split_mode |
Multi-GPU split mode for model load (gh#23 MVP item 6).
Maps to llama.cpp's enum llama_split_mode. Accepted values:
""(default) — keep llama.cpp's default (LAYER)."none"— single GPU, no split. Use withmain_gpu."layer"— split layers + KV across GPUs (llama.cpp default)."row"— split layers + KV with tensor parallelism. Unrecognized values fall back to the default with a logged warning. Empty (default) preserves pre-v2.3.18 model load bit-for-bit.- Version
- 2.3.18
◆ tensor_split
| std::string entropic::ModelConfig::tensor_split |
◆ use_mlock
The documentation for this struct was generated from the following file:
- entropic/types/config.h
