entropic::TokenCounter Class Reference

Track token usage across conversation. More...

#include <entropic/core/compaction.h>

Public Member Functions
	TokenCounter (int max_tokens)
	Construct a token counter.
int	count_message (const Message &msg) const
	Count tokens in a single message.
int	count_messages (const std::vector< Message > &messages) const
	Count total tokens in a message list.
float	usage_percent (const std::vector< Message > &messages) const
	Get usage as fraction of context window (0.0–1.0).
void	clear_cache ()
	Clear the token count cache.

Public Attributes
int	max_tokens
	Maximum context window size.

Detailed Description

Track token usage across conversation.

Uses simple heuristic (~4 chars per token) for estimation. More accurate counting requires the model's tokenizer.

Version

1.8.4

Definition at line 32 of file compaction.h.

Constructor & Destructor Documentation

TokenCounter()

entropic::TokenCounter::TokenCounter ( int  max_tokens )

explicit

Construct a token counter.

Parameters

max_tokens

Maximum context window size.

Version

1.8.4

Parameters

max_tokens

Maximum context window size. @dg_internal

Version

1.8.4

Definition at line 26 of file compaction.cpp.

Member Function Documentation

clear_cache()

void entropic::TokenCounter::clear_cache

(

)

Clear the token count cache.

Version

1.8.4

@dg_internal

Version

1.8.4

Definition at line 98 of file compaction.cpp.

count_message()

int entropic::TokenCounter::count_message

(

const Message &

msg

)

const

Count tokens in a single message.

Parameters

msg	Message to count.

Returns

Estimated token count. @req REQ-COMPACT-001

Version

1.8.4

Parameters

msg	Message to count.

Returns

Estimated token count (content + role overhead). @req REQ-COMPACT-001

Version

1.8.4

Definition at line 50 of file compaction.cpp.

count_messages()

int entropic::TokenCounter::count_messages

(

const std::vector< Message > &

messages

)

const

Count total tokens in a message list.

Parameters

messages

Messages to count.

Returns

Total estimated token count. @req REQ-COMPACT-001

Version

1.8.4

Definition at line 68 of file compaction.cpp.

usage_percent()

float entropic::TokenCounter::usage_percent

(

const std::vector< Message > &

messages

)

const

Get usage as fraction of context window (0.0–1.0).

Get usage as fraction of context window.

Parameters

messages

Messages to measure.

Returns

Usage fraction. @req REQ-COMPACT-001

Version

1.8.4

Parameters

messages

Messages to measure.

Returns

Usage fraction (0.0 if max_tokens is 0). @req REQ-COMPACT-001

Version

1.8.4

Definition at line 84 of file compaction.cpp.

Member Data Documentation

max_tokens

int entropic::TokenCounter::max_tokens

Maximum context window size.

Definition at line 74 of file compaction.h.

---

The documentation for this class was generated from the following files:

• entropic/core/compaction.h
• core/compaction.cpp