Derive a model's placement from its shape and the free VRAM. More...

#include "gguf_metadata.h"
#include "vram_footprint.h"
#include <cstdint>

Include dependency graph for auto_placement.h:

This graph shows which files directly or indirectly include this file:

Classes
struct	entropic::AutoPlacement
	A placement decision, with the reason it was reached. More...

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Functions
uint64_t	entropic::compute_allowance_bytes (const FootprintInputs &base, int n_ubatch)
	VRAM to hold back for graph/activation scratch.

bool	entropic::placement_fits (const FootprintInputs &base, const GgufShape &shape, int gpu_layers, int cpu_moe_layers, uint64_t budget_bytes, uint64_t *used)
	Does a candidate placement fit the budget?

bool	entropic::try_fully_resident (const GgufShape &shape, const FootprintInputs &base, uint64_t budget, AutoPlacement *out)
	Try the best case: every layer and every expert on the card.

bool	entropic::try_expert_offload (const GgufShape &shape, const FootprintInputs &base, uint64_t budget, AutoPlacement *out)
	Try keeping every layer resident by moving experts host-side.

bool	entropic::try_fewer_layers (const GgufShape &shape, const FootprintInputs &base, uint64_t budget, AutoPlacement *out)
	Last resort: take the largest layer count that fits.

bool	entropic::try_smaller_ubatch (const GgufShape &shape, const FootprintInputs &base, uint64_t free_vram_bytes, int configured, AutoPlacement *out)
	Try full residency at each ubatch below the configured one.

AutoPlacement	entropic::derive_auto_placement (const GgufShape &shape, const FootprintInputs &base, uint64_t free_vram_bytes, int n_ubatch=0)
	Choose the placement that uses the card best without overcommitting.

Variables
constexpr uint64_t	entropic::kComputeBytesAtUbatch512 = 1222ull * 1024ull * 1024ull
	Measured compute scratch for ONE context at a 512-token ubatch (vram_footprint.h: 1222 MiB, gemma-4 E4B with an MTP head).

constexpr int	entropic::kComputeReferenceUbatch = 512
	The ubatch that figure was measured at.

constexpr int	entropic::kDefaultUbatch = 512
	llama.cpp's default ubatch, assumed when the tier does not set one.

constexpr int	entropic::kUbatchLadder [] = {512, 256, 128}
	Values auto will step down through, descending.

Detailed Description

Derive a model's placement from its shape and the free VRAM.

gpu_layers: auto exists so that fitting a new model is a decision the engine makes rather than a number an operator tunes. Until v2.13.1 it made that decision badly: partial_gpu_layers_for priced every model as thirty layers, charged file_bytes / 30 per layer — folding in embeddings and the output head, which do not move with gpu_layers — and reserved a flat 2 GiB for everything that is not weights, double-counting against the vram_reserve_mb the real estimator already honours. It knew nothing about the KV cache, the draft head, or experts. On the 26B-A4B at IQ2 it derives 26 of 30 layers for a model measured to run at 30 of 30 with headroom to spare.

This solves the estimator instead of approximating beside it. There is one footprint model in the engine (vram_footprint.h); auto asks it which placements fit and takes the best one.

Order of concessions. When everything does not fit, a MoE gives up its experts before it gives up a layer. Moving one layer off the card takes its attention with it, and attention is what speculative decode and every prefill leans on; moving that layer's experts off frees the bulk of its bytes and leaves the attention resident. That ordering is why v2.13.0 measured expert offload at 22.62 tok/s against 18.86 for whole-layer offload of the same model.

Version: 2.13.1

Definition in file auto_placement.h.

Classes

Namespaces

Functions

Variables

Detailed Description