partial_offload.h File Reference

How many layers of an oversized GGUF fit the free VRAM, purely. More...

#include <algorithm>
#include <cstdint>
#include <cstdlib>

Include dependency graph for partial_offload.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Functions
int	entropic::partial_gpu_layers_for (uint64_t file_bytes, uint64_t free_vram_bytes, int n_layers=30)
	Layers to place on the GPU for an oversized model.
bool	entropic::host_can_hold_warm_load (uint64_t file_bytes, uint64_t available_bytes, uint64_t headroom_bytes=1024ull *1024 *1024)
	Whether the host can hold a model's WARM load.
bool	entropic::large_model_tests_waived (uint64_t file_bytes)
	Whether an operator has explicitly waived one LARGE model's test.
bool	entropic::mlock_refused (bool use_mlock, uint64_t file_bytes, int gpu_layers, uint64_t memlock_limit_bytes, uint64_t host_expert_bytes=0)
	Whether an explicit use_mlock would be fatal for this model.
bool	entropic::gpu_offload_refused (uint64_t file_bytes, int gpu_layers, uint64_t free_vram_bytes)
	Whether a requested GPU offload provably cannot fit free VRAM.

Variables
constexpr uint64_t	entropic::kPartialOffloadReserveBytes = 2048ull * 1024 * 1024
	VRAM held back for compute/graph buffers, which scale with ubatch and model internals rather than with layer count.
constexpr uint64_t	entropic::kLargeModelThresholdBytes = 10ull * 1024 * 1024 * 1024
	Size above which a model counts as "large" for the waiver below.
constexpr uint64_t	entropic::kMemlockUnlimited = UINT64_MAX
	Sentinel for an unlimited RLIMIT_MEMLOCK (gh#148).

Detailed Description

How many layers of an oversized GGUF fit the free VRAM, purely.

A model larger than the card runs partially offloaded: some layers on the GPU, the rest in system RAM. Choosing that split badly is how a "slow but working" configuration becomes an OOM, and the direction is easy to invert — every layer moved OFF the GPU ADDS to the host footprint, so more CPU offload makes memory pressure worse, not better.

The v2.7.0 accommodation hardcoded 15 layers from a measurement taken while the desktop held ~1.8 GB of VRAM. It never adapted, so on a quiet card reporting 10.2 GB free it still left ~8.2 GB of a 13.2 GB model in RAM.

Pure and vendor-free so the arithmetic is CPU-unit-testable with no GPU and no model — the same reason warm_keep_util.h and session_pool_util.h are.

Version

2.12.0

Definition in file partial_offload.h.