Read a GGUF's shape without loading its tensors (v2.13.1). More...
#include <cstdint>#include <string>
Go to the source code of this file.
Classes | |
| struct | entropic::GgufShape |
| A GGUF's shape, as far as its metadata states it. More... | |
Namespaces | |
| namespace | entropic |
| Activate model on GPU (WARM → ACTIVE). | |
Functions | |
| GgufShape | entropic::read_gguf_shape (const std::string &path) |
| Read a GGUF's shape from its metadata, without loading tensor data. | |
Detailed Description
Read a GGUF's shape without loading its tensors (v2.13.1).
gpu_layers: auto has to answer "how many layers fit" before anything is loaded, and until v2.13.1 it answered without knowing how many layers the model had. partial_offload.h deliberately reads no GGUF metadata — a pure header with no llama.cpp dependency — so it defaults to 30 layers and documents that as a low-end guess. That guess is exactly right for the 26B-A4B and 40% wrong for a 42-layer E4B, which is priced as if each layer cost file_bytes / 30 and therefore derives a count that leaves the card underused (gh#192).
The information was always available. gguf_init_from_file with no_alloc = true maps the header, the key/value metadata and the tensor index, and reads none of the tensor data — so a 10 GB model costs a header read to answer:
general.architecture, which names the other keys<arch>.block_count, the real layer count<arch>.expert_count, non-zero exactly for a MoE- every tensor's name and size, which is what makes expert placement EXACT rather than a guessed fraction of a layer
That last point is the one that matters for a MoE. Deriving an expert split from "experts are most of a layer" would be a heuristic deciding residency; summing the bytes of the tensors that actually match the expert pattern is a measurement.
- Version
- 2.13.1
Definition in file gguf_metadata.h.
