inference Directory Reference

Directory dependency graph for inference:

Directories
	adapters

Files
	adapter_manager.cpp
	AdapterManager implementation — LoRA lifecycle and hot-swap.
	auto_placement.h
	Derive a model's placement from its shape and the free VRAM.
	backend.cpp
	InferenceBackend base class implementation.
	batch_kv_util.h
	gh#158: which sequences a same-prefix batch may touch, purely.
	batch_util.h
	gh#98 (v2.8.0) same-prefix batch-generation decision logic.
	context_fit.h
	v2.13.0 prompt-vs-context admission arithmetic for one turn.
	device_memory.cpp
	gh#142: ggml-backed free-VRAM query.
	device_memory.h
	gh#142: ask the GPU how much VRAM is actually free.
	empty_content_diagnosis.h
	gh#137: explain an empty turn without misattributing its cause.
	expert_offload.h
	EXPERIMENTAL (v2.13.0): put a MoE layer's routed-expert tensors on the host while its attention, KV and dense weights stay on the card.
	expert_offload_buft.h
	EXPERIMENTAL (v2.13.0): the llama.cpp binding for expert offload — an owning, null-terminated llama_model_tensor_buft_override array.
	generation_summary.h
	One post-generation summary line, formatted identically on every decode path.
	gguf_metadata.cpp
	Metadata-only GGUF reads for the residency estimator (v2.13.1).
	gguf_metadata.h
	Read a GGUF's shape without loading its tensors (v2.13.1).
	grammar_registry.cpp
	GrammarRegistry implementation — named grammar management.
	grammar_source.h
	Which grammar source constrains a decode (gh#134).
	image_preprocessor.cpp
	Image preprocessing implementation.
	inference_c_api.cpp
	C API wrappers for InferenceBackend.
	interface_factory.cpp
	InferenceInterface factory — bridges orchestrator to C callbacks.
	llama_cpp_backend.cpp
	LlamaCppBackend implementation — direct llama.cpp C API.
	llama_cpp_backend.h
	LlamaCppBackend — llama.cpp C API integration.
	llama_cpp_sampler.cpp
	LlamaCppSampler + LlamaCppSamplerFactory implementation (v2.3.10 seam impl).
	llama_cpp_sampler.h
	Concrete llama.cpp Sampler + SamplerFactory (v2.3.10 seam impl).
	llama_cpp_tokenizer.cpp
	LlamaCppTokenizer implementation (v2.3.10 seam impl).
	llama_cpp_tokenizer.h
	Concrete llama.cpp tokenizer (v2.3.10 seam impl).
	mtp_envelope.h
	Pure envelope check for the MTP speculative path (gh#108).
	orchestrator.cpp
	ModelOrchestrator implementation.
	partial_offload.h
	How many layers of an oversized GGUF fit the free VRAM, purely.
	profile_registry.cpp
	ProfileRegistry implementation – named GPU resource profiles.
	prompt_cache.cpp
	PromptCache implementation — LRU eviction, thread-safe.
	prompt_cache.h
	Host-memory KV cache state storage with LRU eviction.
	response_parse.cpp
	Template-first / adapter-second parse rule (gh#108).
	response_parse.h
	One template-first / adapter-second parse rule for raw model output.
	secondary_model_loader.cpp
	SecondaryModelLoader implementation.
	session_pool_util.h
	gh#144: derive a tier's KV geometry from max_sessions, purely.
	session_residency.h
	gh#144: which llama sequence holds which session's KV, purely.
	speculative_compat.cpp
	Implementation of the target/draft compatibility check.
	throughput_tracker.cpp
	ThroughputTracker implementation – EWMA throughput measurement.
	tool_call_markers.h
	gh#103 (v2.8.2): family-aware tool-call CLOSE markers, derived from the resolved common_chat format.
	tool_call_serialize.h
	Shared serialization of parsed tool calls to the C-ABI JSON array form.
	vram_footprint.h
	gh#142: a VRAM footprint estimate honest enough to refuse a load on.
	warm_keep_util.h
	gh#96 (v2.7.5) warm-keep / incremental-prefill decision logic.