Everything the estimate needs, with no orchestrator or filesystem. More...

#include </home/runner/work/entropic/entropic/src/inference/vram_footprint.h>

Public Attributes
uint64_t	weights_bytes = 0
	Size of the tier's GGUF on disk.

uint64_t	mmproj_bytes = 0
	Size of the vision projector, 0 if none.

int	gpu_layers = -1
	Tier's requested offload (-1 = all).

int	context_length = 0
	Tier's requested context window.

std::string	cache_type_k = "f16"
	KV key cache quantization.

std::string	cache_type_v = "f16"
	KV value cache quantization.

int	vram_reserve_mb = 0
	Configured headroom to leave free.

int	max_sessions = 1
	Resident conversation sessions this tier keeps KV for (gh#144).

uint64_t	draft_bytes = 0
	Draft / MTP head bytes, resident alongside the target.

GGUF-derived shape (v2.13.1)
Zero means "not read", and the estimator then behaves exactly as it did before — a partial offload stays unpriceable. Supplied, they let a partial offload be priced EXACTLY, which is what previously forced `partial_unknown` and left the budget gate open (gh#142).
int	block_count = 0
	Real layer count from `<arch>.block_count`.

uint64_t	block_bytes = 0
	Total bytes of per-block tensors.

uint64_t	non_block_bytes = 0
	Embeddings/output/final norms.

uint64_t	host_expert_bytes = 0
	Expert bytes deliberately kept host-side by `cpu_moe_layers`.

Detailed Description

Everything the estimate needs, with no orchestrator or filesystem.

Definition at line 83 of file vram_footprint.h.

Member Data Documentation

◆ block_bytes

uint64_t entropic::FootprintInputs::block_bytes = 0

Total bytes of per-block tensors.

Definition at line 119 of file vram_footprint.h.

◆ block_count

int entropic::FootprintInputs::block_count = 0

Real layer count from <arch>.block_count.

Definition at line 118 of file vram_footprint.h.

◆ cache_type_k

std::string entropic::FootprintInputs::cache_type_k = "f16"

KV key cache quantization.

Definition at line 88 of file vram_footprint.h.

◆ cache_type_v

std::string entropic::FootprintInputs::cache_type_v = "f16"

KV value cache quantization.

Definition at line 89 of file vram_footprint.h.

◆ context_length

int entropic::FootprintInputs::context_length = 0

Tier's requested context window.

Definition at line 87 of file vram_footprint.h.

◆ draft_bytes

uint64_t entropic::FootprintInputs::draft_bytes = 0

Draft / MTP head bytes, resident alongside the target.

v2.13.1: priced nowhere before. A target-owned MTP head is small (225 MiB for mtp_a4b) but it is loaded onto the same card, and on an 11 GiB device against a 9.3 GiB trunk it is the difference between fitting and not.

Version: 2.13.1

Definition at line 109 of file vram_footprint.h.

◆ gpu_layers

int entropic::FootprintInputs::gpu_layers = -1

Tier's requested offload (-1 = all).

Definition at line 86 of file vram_footprint.h.

◆ host_expert_bytes

uint64_t entropic::FootprintInputs::host_expert_bytes = 0

Expert bytes deliberately kept host-side by cpu_moe_layers.

Subtracted from the resident weight term: this is what an expert split actually frees on the card, summed from the tensors that match llama.cpp's expert pattern rather than guessed as a fraction.

Definition at line 127 of file vram_footprint.h.

◆ max_sessions

int entropic::FootprintInputs::max_sessions = 1

Resident conversation sessions this tier keeps KV for (gh#144).

The KV term multiplies by this. context_length is PER SESSION, so a pool of 3 at 32768 allocates 98304 cells. Without this field the gate under-counts a pool by exactly N and can admit a configuration that then aborts the host process inside llama.cpp — the precise failure gh#142 built this gate to prevent.

Version: 2.12.0

Definition at line 100 of file vram_footprint.h.

◆ mmproj_bytes

uint64_t entropic::FootprintInputs::mmproj_bytes = 0

Size of the vision projector, 0 if none.

Definition at line 85 of file vram_footprint.h.

◆ non_block_bytes

uint64_t entropic::FootprintInputs::non_block_bytes = 0

Embeddings/output/final norms.

Definition at line 120 of file vram_footprint.h.

◆ vram_reserve_mb

int entropic::FootprintInputs::vram_reserve_mb = 0

Configured headroom to leave free.

Definition at line 90 of file vram_footprint.h.

◆ weights_bytes

uint64_t entropic::FootprintInputs::weights_bytes = 0

Size of the tier's GGUF on disk.

Definition at line 84 of file vram_footprint.h.

The documentation for this struct was generated from the following file:

inference/vram_footprint.h

Public Attributes