expert_offload_buft.h File Reference

EXPERIMENTAL (v2.13.0): the llama.cpp binding for expert offload — an owning, null-terminated llama_model_tensor_buft_override array. More...

#include "expert_offload.h"
#include <llama.h>
#include <cstddef>
#include <string>
#include <vector>

Include dependency graph for expert_offload_buft.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	entropic::ExpertOffloadOverrides
	Owning, null-terminated tensor-buffer override list for expert offload. More...

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Detailed Description

EXPERIMENTAL (v2.13.0): the llama.cpp binding for expert offload — an owning, null-terminated llama_model_tensor_buft_override array.

llama_model_params::tensor_buft_overrides is a raw pointer to a null-terminated array whose elements hold const char* pattern. llama.cpp copies neither: the loader stores the pointer and dereferences the strings while it places tensors (llama-model-loader.cpp, std::regex_search per tensor per pattern). The array AND every string it points at must therefore outlive llama_model_load_from_file.

Upstream solves this with a function-local static std::list<std::string> that is never freed (llm_add_n_cpu_ffn_overrides). That is fine for a CLI that loads one model and exits; it is not fine for a library that loads, unloads and reloads models across many handles, so this owns its storage instead and dies with the scope that built it. The invariant is the one property worth asserting in a unit test: patterns_ is filled ONCE and never touched again, so the c_str() values stored in overrides_ stay valid for the object's whole life.

Copy and move are deleted — not because a move would be unsafe (a std::vector move transfers the buffer; the std::string elements themselves do not move) but because "this object contains pointers into itself" is a claim better enforced than reasoned about. Construct it in the scope that performs the load.

Separated from expert_offload.h so that the pure rule stays vendor-free and src/config/validate.cpp — which links no llama.cpp — can include it.

Version

2.13.0

Definition in file expert_offload_buft.h.