entropic::LlamaCppTokenizer Class Reference

Tokenizer adapter that forwards to llama.cpp's vocab API. More...

#include </home/runner/work/entropic/entropic/src/inference/llama_cpp_tokenizer.h>

Inheritance diagram for entropic::LlamaCppTokenizer:

Collaboration diagram for entropic::LlamaCppTokenizer:

Public Member Functions
	LlamaCppTokenizer (const llama_vocab *vocab)
	Construct with a borrowed vocab pointer.
std::vector< int32_t >	tokenize (const std::string &text, bool add_special) const override
	Encode text into token ids via the wrapped vocab.
std::string	detokenize (int32_t token) const override
	Decode a single token id to its surface string.
Public Member Functions inherited from entropic::Tokenizer
virtual int	count_tokens (const std::string &text) const
	Count tokens in text without retaining the IDs.

Detailed Description

Tokenizer adapter that forwards to llama.cpp's vocab API.

Borrowing semantics: vocab pointer is owned by the model. Caller (LlamaCppBackend) must ensure the model outlives the tokenizer.

Definition at line 35 of file llama_cpp_tokenizer.h.

Constructor & Destructor Documentation

LlamaCppTokenizer()

entropic::LlamaCppTokenizer::LlamaCppTokenizer ( const llama_vocab *  vocab )

explicit

Construct with a borrowed vocab pointer.

Construct a tokenizer borrowing a llama_vocab pointer.

Parameters

vocab

Non-owning pointer; must outlive this tokenizer.

Version

2.3.10

vocab is borrowed, not owned — its lifetime must outlive this object. LlamaCppBackend resets the tokenizer before freeing the backing llama_model so the borrow never dangles.

Definition at line 37 of file llama_cpp_tokenizer.cpp.

Member Function Documentation

detokenize()

std::string entropic::LlamaCppTokenizer::detokenize ( int32_t  token ) const

overridevirtual

Decode a single token id to its surface string.

special=false — special tokens (BOS, EOS, channel markers) do not render to surface text. History (gh#68, gh#65): defensive flag is kept regardless of whether the current model fleet emits special tokens, since the seam is shared across families.

Parameters

token

Token id to decode.

Returns

Surface string. Empty when vocab_ is null or the retry decode returned non-positive.

Implements entropic::Tokenizer.

Definition at line 86 of file llama_cpp_tokenizer.cpp.

tokenize()

std::vector< int32_t > entropic::LlamaCppTokenizer::tokenize ( const std::string &  text, bool  add_special  ) const

overridevirtual

Encode text into token ids via the wrapped vocab.

Parameters

text	Input string (any UTF-8).
add_special	True to prepend BOS / model-defined special tokens.

Returns

Token id vector. Empty if vocab_ is null or llama_tokenize returned a negative actual-count on the sized retry.

Implements entropic::Tokenizer.

Definition at line 49 of file llama_cpp_tokenizer.cpp.

---

The documentation for this class was generated from the following files:

• inference/llama_cpp_tokenizer.h
• inference/llama_cpp_tokenizer.cpp