tokenizer.h File Reference

Abstract Tokenizer seam for backend testability (v2.3.10). More...

#include <cstdint>
#include <string>
#include <vector>

Include dependency graph for tokenizer.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
class	entropic::Tokenizer
	Pure-virtual tokenizer surface used by the inference backend. More...

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Detailed Description

Abstract Tokenizer seam for backend testability (v2.3.10).

Pre-v2.3.10 LlamaCppBackend made llama.cpp tokenizer calls inline — meaning the tokenization paths were only exercisable with a loaded model. The unit-test coverage gate (CPU-only, no GGUFs) could never reach those lines. This abstract seam lets the backend's tokenize/detokenize/count callers route through a mockable interface; the production backend wires a LlamaCppTokenizer that forwards to llama.cpp; unit tests inject MockTokenizer and assert behavior without a model.

Version

2.3.10

Definition in file tokenizer.h.