entropic::Tokenizer Class Reference

Pure-virtual tokenizer surface used by the inference backend. More...

#include <entropic/inference/tokenizer.h>

Inheritance diagram for entropic::Tokenizer:

Public Member Functions
virtual std::vector< int32_t >	tokenize (const std::string &text, bool add_special) const =0
	Encode text to token IDs.
virtual std::string	detokenize (int32_t token) const =0
	Decode a single token to its surface piece.
virtual int	count_tokens (const std::string &text) const
	Count tokens in text without retaining the IDs.

Detailed Description

Pure-virtual tokenizer surface used by the inference backend.

Three responsibilities:

1. Encode text → token ids (with optional BOS/EOS specials).
2. Decode a single token → string piece.
3. Count tokens for capacity-planning (e.g., context budget).

Implementations may be lossy (count is exact when model is loaded, estimate otherwise) but must never throw and must never crash on empty inputs. Callers treat returned vectors as authoritative.

Version

2.3.10

Definition at line 40 of file tokenizer.h.

Member Function Documentation

count_tokens()

virtual int entropic::Tokenizer::count_tokens ( const std::string &  text ) const

inlinevirtual

Count tokens in text without retaining the IDs.

Default implementation invokes tokenize() and returns size(). Overrides may use a faster path (e.g., model size-prediction without actual allocation).

Parameters

text	UTF-8 input.

Returns

Token count (>= 0). Returns 0 on empty input.

Definition at line 74 of file tokenizer.h.

detokenize()

virtual std::string entropic::Tokenizer::detokenize ( int32_t  token ) const

pure virtual

Decode a single token to its surface piece.

Parameters

token

Token ID.

Returns

Surface string ("" for special tokens that don't render).

Version

2.3.10

Implemented in entropic::LlamaCppTokenizer.

tokenize()

virtual std::vector< int32_t > entropic::Tokenizer::tokenize ( const std::string &  text, bool  add_special  ) const

pure virtual

Encode text to token IDs.

Parameters

text	UTF-8 input.
add_special	Insert model-specific BOS/EOS specials when true.

Returns

Token IDs (empty on failure or empty input).

Version

2.3.10

Implemented in entropic::LlamaCppTokenizer.

---

The documentation for this class was generated from the following file:

• entropic/inference/tokenizer.h