expert_offload.h File Reference

EXPERIMENTAL (v2.13.0): put a MoE layer's routed-expert tensors on the host while its attention, KV and dense weights stay on the card. More...

#include <entropic/types/config.h>
#include <string>
#include <vector>

Include dependency graph for expert_offload.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Functions
std::string	entropic::expert_block_pattern (int layer)
	The override pattern for ONE layer's expert tensors.
std::vector< std::string >	entropic::expert_offload_patterns (int cpu_moe_layers)
	Patterns for the FIRST cpu_moe_layers layers, in layer order.
std::string	entropic::expert_offload_conflict_reason (const ModelConfig &cfg)
	Configuration-time refusal for a combination that cannot mean what it says.
std::string	entropic::expert_offload_load_refusal (int cpu_moe_layers, int n_layer, int n_expert)
	Post-load refusal, against the numbers only the loaded model knows.

Variables
constexpr const char *	entropic::kExpertTensorRegex
	Tensor-name pattern tail identifying a layer's ROUTED-expert weights.

Detailed Description

EXPERIMENTAL (v2.13.0): put a MoE layer's routed-expert tensors on the host while its attention, KV and dense weights stay on the card.

Why this exists

gpu_layers is role-blind. A layer is either wholly on the device or wholly on the host, so a model that does not fit spends VRAM on rarely touched expert weights and exiles attention — and its KV — to system RAM. Measured at v2.13.0 (gh#153, decision #42): Gemma 4 26B-A4B QAT (14.25 GB) on an 11 GiB GTX 1080 Ti resolves gpu_layers: auto to 18 of 31 layers and runs at 18.7 tok/s, and MTP — worth +42.8 % on the fully resident E4B — is worth nothing there (−0.92 % / +0.82 % across two runs against a same-config floor of 0.5–1.0 %). Decision #42 item (iii) records that the split "moves WHOLE layers, attention included, and says nothing about an experts-only split". This is the knob that lets someone measure the other split.

Published practice is consistent about the direction — KTransformers, Fiddler (ICLR'25) and mixtral-offloading all keep attention, KV and the shared/dense path resident and push only routed-expert FFN weights to the host. llama.cpp exposes exactly that through llama_model_params::tensor_buft_overrides (-ncmoe / --n-cpu-moe), which is the mechanism this header mirrors rather than reinvents.

Why this header is pure

Same reason as partial_offload.h and session_pool_util.h: the rule is string construction plus a handful of integer refusals, and it must be assertable by a CPU unit test with no GPU, no GGUF and no llama.cpp. The binding to llama_model_tensor_buft_override lives next door in expert_offload_buft.h, which is the only part that needs the vendor headers — and which validate.cpp therefore does not have to include.

Warning

PROTOTYPE, default off. As of v2.13.1 gpu_layers: auto DOES model expert placement — it reads the real expert-tensor bytes from GGUF metadata and prefers an expert split over dropping a layer. The two remain mutually exclusive, but for the opposite reason: auto derives the split itself, so setting both asks two things to decide one placement.

Version

2.13.0

Definition in file expert_offload.h.