llama_cpp_tokenizer.cpp File Reference

LlamaCppTokenizer implementation (v2.3.10 seam impl). More...

#include "llama_cpp_tokenizer.h"
#include <entropic/types/logging.h>
#include <llama.h>
#include <vector>

Include dependency graph for llama_cpp_tokenizer.cpp:

Go to the source code of this file.

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Variables
static auto	logger = entropic::log::get("inference.tokenizer")

Detailed Description

LlamaCppTokenizer implementation (v2.3.10 seam impl).

Forwards to llama_tokenize / llama_token_to_piece. This file is uncovered by the CPU-only unit-test gate by design — its only job is to be a thin pass-through to llama.cpp. Coverage of the LOGIC that consumes these results lives in LlamaCppBackend tests (which mock the Tokenizer interface).

Definition in file llama_cpp_tokenizer.cpp.

Variable Documentation

logger

auto logger = entropic::log::get("inference.tokenizer")

static

Definition at line 24 of file llama_cpp_tokenizer.cpp.