gh#142: a VRAM footprint estimate honest enough to refuse a load on. More...

#include <cstdint>
#include <string>
#include <utility>

Include dependency graph for vram_footprint.h:

This graph shows which files directly or indirectly include this file:

Classes
struct	entropic::FootprintInputs
	Everything the estimate needs, with no orchestrator or filesystem. More...

struct	entropic::FootprintEstimate
	An estimate, or an explicit admission that there isn't one. More...

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Enumerations
enum class	entropic::Offload { entropic::none , entropic::full , entropic::partial_unknown }
	Where a tier's weights land, as far as the estimate can tell. More...

Functions
double	entropic::kv_scale_for_cache_type (const std::string &type)
	Cost of a KV cache type relative to f16.

Offload	entropic::classify_offload (int gpu_layers)
	Classify a tier's offload request into a priceable placement.

double	entropic::kv_bytes_per_token (const FootprintInputs &in)
	Per-token KV cost in bytes for this tier's cache configuration.

FootprintEstimate	entropic::estimate_vram_footprint (const FootprintInputs &in)
	Estimate the VRAM a tier will occupy, or report that it cannot.

int	entropic::recommend_context_length (const FootprintInputs &in, uint64_t available_bytes)
	Largest context length that fits, for the "won't fit" recommendation.

Variables
constexpr uint64_t	entropic::kBaseKvPerTokenF16 = 16ull * 1024ull
	KV bytes per token at f16, the rate the v2.2.4 estimate assumed.

constexpr int	entropic::kAllLayersSentinel = 99
	`gpu_layers` at or above this means "every layer" (llama.cpp's 99).

Detailed Description

gh#142: a VRAM footprint estimate honest enough to refuse a load on.

Why this exists: The v2.2.4 admission gate (REQ-INFER-019) compared a tier's estimated footprint against a VRAM budget and refused with ENTROPIC_ERROR_TIER_MODEL_TOO_LARGE. It was never reached in practice: the budget came only from ENTROPIC_VRAM_BUDGET_BYTES, so on any deployment that did not set that variable the budget was 0, the gate was disabled, and a failed allocation aborted the host process inside llama.cpp (ggml-backend.cpp:179: GGML_ASSERT(buffer)) with no diagnostic.

Switching the gate on required fixing the estimate first, because the old one was wrong in three ways that all bias toward REFUSING configurations that work:

it counted the entire weights file no matter how many layers were actually offloaded, so a partially offloaded model was priced as if fully resident. Qwen3.6-35B-A3B IQ3_XXS (~13 GB) runs at gpu_layers=15 on an 11 GB card; a gate built on that estimate would refuse it.
it priced KV at a flat 16 KiB/token regardless of cache_type, roughly 4x over-counting a q4_0 cache.
it ignored the vision projector, which is the allocation that actually failed in the gh#142 abort.

What this does NOT count — compute buffers: llama.cpp reserves graph/activation scratch per context at load time, sized by ubatch and model internals rather than by context length. Measured on this repo's own benchmark: 1222 MiB for a gemma-4 E4B MTP head's context, at a 512-token ubatch. That is more than twice the default vram_reserve_mb of 512, and a speculative configuration pays it TWICE because it holds two contexts.

It is not counted here because it cannot be derived from anything this header is willing to read — it needs the model's hidden size and layer count, i.e. GGUF metadata. vram_reserve_mb is the knob that must cover it, and callers running near the edge should raise it rather than trust the default.

The consequence is honest and worth stating plainly: this estimate can admit a configuration that then fails to load. It is designed to prevent the catastrophic case (an abort that kills the host process) and to produce an actionable recommendation — not to guarantee a load succeeds. A load that fails after admission surfaces as a typed error, which is the outcome gh#142 asked for.

The rule this file follows: An estimate that cannot be bounded is reported as UNKNOWN rather than guessed. An unknown estimate leaves the gate open — the engine declines to refuse what it cannot price. Refusing a working configuration is a worse failure than missing one that would have failed, because the operator has no way to tell a false refusal from a real one.

Kept pure and free of vendor types so every case is CPU-unit-testable; the device query that supplies available lives in device_memory.h.

Version: 2.11.0

Definition in file vram_footprint.h.

Classes

Namespaces

Enumerations

Functions

Variables

Detailed Description