llama_cpp_tokenizer.h File Reference

Concrete llama.cpp tokenizer (v2.3.10 seam impl). More...

#include <entropic/inference/tokenizer.h>

Include dependency graph for llama_cpp_tokenizer.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	entropic::LlamaCppTokenizer
	Tokenizer adapter that forwards to llama.cpp's vocab API. More...

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Detailed Description

Concrete llama.cpp tokenizer (v2.3.10 seam impl).

Implements the abstract Tokenizer interface against a borrowed llama_vocab* (owned by the model). Lifetime: must be released before the owning model is freed; LlamaCppBackend handles that ordering. Tokenizer methods never throw; on llama.cpp error they log and return empty/0.

Definition in file llama_cpp_tokenizer.h.