entropic::ExpertOffloadOverrides Class Reference

Owning, null-terminated tensor-buffer override list for expert offload. More...

#include </home/runner/work/entropic/entropic/src/inference/expert_offload_buft.h>

Public Member Functions
	ExpertOffloadOverrides (int cpu_moe_layers, ggml_backend_buffer_type_t cpu_buft)
	Build the override list for the first N layers.
	ExpertOffloadOverrides (const ExpertOffloadOverrides &)=delete
ExpertOffloadOverrides &	operator= (const ExpertOffloadOverrides &)=delete
	ExpertOffloadOverrides (ExpertOffloadOverrides &&)=delete
ExpertOffloadOverrides &	operator= (ExpertOffloadOverrides &&)=delete
const llama_model_tensor_buft_override *	data () const
	Pointer to hand to llama_model_params::tensor_buft_overrides.
const std::vector< std::string > &	patterns () const
	The owned pattern strings, in layer order.
std::size_t	size () const
	Number of override entries, excluding the terminator.

Detailed Description

Owning, null-terminated tensor-buffer override list for expert offload.

Empty (and data() == nullptr) whenever cpu_moe_layers <= 0, which is what makes an absent config key byte-identical to every release before this one: llama_model_default_params() already leaves tensor_buft_overrides null, and assigning null over null changes nothing.

@req REQ-INFER-027

Version

2.13.0

Definition at line 59 of file expert_offload_buft.h.

Constructor & Destructor Documentation

ExpertOffloadOverrides()

entropic::ExpertOffloadOverrides::ExpertOffloadOverrides ( int  cpu_moe_layers, ggml_backend_buffer_type_t  cpu_buft  )

inline

Build the override list for the first N layers.

Parameters

cpu_moe_layers	Layers whose expert tensors go to the host; <= 0 builds an empty list.
cpu_buft	Destination buffer type, normally ggml_backend_cpu_buffer_type(). Taken as a parameter rather than called here so the construction is assertable without a live ggml backend registry.

Version

2.13.0

Definition at line 71 of file expert_offload_buft.h.

Member Function Documentation

data()

const llama_model_tensor_buft_override * entropic::ExpertOffloadOverrides::data ( ) const

inline

Pointer to hand to llama_model_params::tensor_buft_overrides.

Returns

Null-terminated array, or nullptr when the feature is off. @req REQ-INFER-027

Version

2.13.0

Definition at line 97 of file expert_offload_buft.h.

patterns()

const std::vector< std::string > & entropic::ExpertOffloadOverrides::patterns ( ) const

inline

The owned pattern strings, in layer order.

Returns

Stable storage backing every pattern in data(). @utility

Version

2.13.0

Definition at line 107 of file expert_offload_buft.h.

size()

std::size_t entropic::ExpertOffloadOverrides::size ( ) const

inline

Number of override entries, excluding the terminator.

Returns

Layer count actually overridden; 0 when off. @utility

Version

2.13.0

Definition at line 115 of file expert_offload_buft.h.

---

The documentation for this class was generated from the following file:

• inference/expert_offload_buft.h