prompt_cache.h File Reference

Host-memory KV cache state storage with LRU eviction. More...

#include <cstddef>
#include <cstdint>
#include <list>
#include <mutex>
#include <string>
#include <string_view>
#include <unordered_map>
#include <vector>

Include dependency graph for prompt_cache.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
struct	entropic::CacheKey
	64-bit hash used as cache lookup key. More...
struct	entropic::CacheKeyHash
	Hash function for CacheKey in unordered containers. More...
struct	entropic::CacheEntry
	Single cached KV state snapshot. More...
struct	entropic::CacheStats
	Cumulative cache performance counters. More...
class	entropic::PromptCache
	Host-memory KV cache with LRU eviction. More...

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Detailed Description

Host-memory KV cache state storage with LRU eviction.

Caches processed system prompt prefixes (identity + constitution + tools) in host RAM. On tier swap, restores cached prefix via llama_state_seq_set_data instead of re-processing prompt tokens.

Internal to librentropic-inference. Does not cross .so boundaries. Thread-safe: all public methods acquire the internal mutex.

Version

1.8.3

Definition in file prompt_cache.h.