utf8_sanitize.h File Reference

UTF-8 validation + replacement at every system boundary where bytes change ownership. More...

#include <entropic/entropic_export.h>
#include <string>
#include <string_view>

Include dependency graph for utf8_sanitize.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Functions
ENTROPIC_EXPORT std::string	entropic::mcp::sanitize_utf8 (std::string_view input)
	Replace invalid UTF-8 byte sequences with U+FFFD.

Detailed Description

UTF-8 validation + replacement at every system boundary where bytes change ownership.

Bytes carrying invalid UTF-8 (legacy-codepage source comments, mojibake, truncated multi-byte runs, model-stream desyncs under XML-tool-call pressure) crash any downstream nlohmann::json::dump() with json::type_error 316. v2.1.0 introduced the sanitizer at the MCP inbound boundary on the assumption that "sanitize once at inbound, trust downstream" was sufficient. v2.1.1 (issue #3) generalized that to a boundary-of-ownership policy after the demo session showed bytes reaching json::dump() from paths that bypassed the inbound boundary.

Policy: sanitize at every external boundary where bytes change

ownership.

• Inbound from MCP servers → src/mcp/tool_executor.cpp
• Inbound from llama_cpp stream → src/core/response_generator.cpp (sanitize once at message-finalization, NEVER per-token — a multi-byte codepoint can split across token boundaries)
• Inbound from tool-call parse → src/core/engine.cpp (AgentEngine::parse_tool_calls — a SEPARATE channel from the content sanitize above. The backend re-derives *cleaned and *tool_calls_json from its own parse of the RAW generation, so a split multi-byte codepoint survives here even after the content string was sanitized. Both feed downstream json::dump() — the cleaned content becomes an assistant message / delegation-summary fallback, the tool args become CompleteTool / directive JSON. v2.9.8, gh#111 recurrence)
• Inbound from audit-log files → src/facade/entropic_audit.cpp
• Inbound from hook plugins → src/core/engine.cpp (fire_post_generate_hook, fire_complete_hook) and src/mcp/tool_executor.cpp (fire_post_tool_hook) — a hook's revised/returned text crosses a plugin .so boundary the same way an MCP tool result does. Missed by the original v2.1.1 pass; fixed as a gh#3 recurrence in v2.9.7, gh#111)
• Outbound to C-API consumers → src/facade/json_serializers.h

Interior code (engine memory, dedup cache, per-tier routing) trusts the bytes — once a string has crossed an inbound boundary, no further sanitize call is needed. Conversely, do NOT add sanitize calls inside loops or hot interior paths; they belong at the outermost system seam each direction. Hook contexts are NOT interior — a registered hook is a plugin .so, i.e. an external boundary in both directions (see gh#3 recurrence above).

sanitize_utf8() walks the input as a byte sequence, validates each codepoint per RFC 3629, and writes U+FFFD (REPLACEMENT CHARACTER, 0xEF 0xBF 0xBD) in place of any malformed sequence. ASCII and well-formed multi-byte input pass through verbatim. The namespace is entropic::mcp for legacy reasons (v2.1.0 introduced it as an MCP-only utility); relocation to a generic namespace would change the exported C++ symbol and is deferred to a future major release.

Version

2.1.1

Definition in file utf8_sanitize.h.

Function Documentation

sanitize_utf8()

std::string entropic::mcp::sanitize_utf8

(

std::string_view

input

)

Replace invalid UTF-8 byte sequences with U+FFFD.

Parameters

input

Raw bytes from a tool-result subprocess. Treated as a byte sequence, not a code-point sequence.

Returns

A new string equal to input if already valid UTF-8, or with each malformed byte sequence replaced by U+FFFD (the Unicode replacement character).

Algorithm:

Per RFC 3629:

• 0x00..0x7F → 1-byte ASCII, pass through
• 0xC2..0xDF + 1 cont → 2-byte sequence
• 0xE0..0xEF + 2 cont → 3-byte sequence (with sub-range checks on the first continuation byte)
• 0xF0..0xF4 + 3 cont → 4-byte sequence (with sub-range checks)
• anything else → replace with U+FFFD, advance one byte

Continuation bytes must be in 0x80..0xBF; a missing or out-of-range continuation triggers replacement and advances past the leading byte only (the next byte gets a fresh validation pass — Bjoern Hoehrmann's "robust resync" property).

@utility

Version

2.1.0

Applied at every inbound boundary that admits untrusted bytes — MCP server output, hook-transformed results, model text — so nothing downstream has to defend against malformed UTF-8.

Parameters

input

Raw bytes (potentially malformed).

Returns

A string equal to the input when it is already valid UTF-8; otherwise the same bytes with each invalid sequence replaced by U+FFFD, one replacement per offending byte. @req REQ-SAFE-001

Version

2.1.0

Definition at line 103 of file utf8_sanitize.cpp.