gh#144: which llama sequence holds which session's KV, purely. More...
#include <cstddef>#include <cstdint>#include <string>#include <unordered_map>#include <vector>
Go to the source code of this file.
Classes | |
| class | entropic::SessionResidency< Tok > |
| Per-session llama sequence slots and their resident tokens. More... | |
Namespaces | |
| namespace | entropic |
| Activate model on GPU (WARM → ACTIVE). | |
Variables | |
| constexpr int | entropic::kNoSessionSlot = -1 |
| Sentinel for "no slot assigned". | |
Detailed Description
gh#144: which llama sequence holds which session's KV, purely.
- Why this exists
- Warm-keep decides reuse by prefix-matching against ONE
resident_tokens_vector, hardcoded to sequence 0. With a single shared conversation that is correct: the history is one monotonically growing sequence, so the previous turn is always a prefix of the next.
Keying conversations per caller breaks that assumption in the worst direction. Two sessions on one handle share a system prompt, so common_prefix_len is greater than zero, so warm_keep_cut returns a NON-ZERO cut and try_warm_reuse takes the REUSE branch — seq_rm destroying the other session's tail and, because it then returns true, skipping the prompt cache as well. Worse than the pre-gh#96 path. The in-repo comment claiming that case "falls back" was wrong; neither stated branch fires (corrected in the same release).
So residency must become per-sequence BEFORE keying ships. This header owns the mapping and the per-slot token vectors; the backend owns the llama calls. Pure and vendor-free so every rule is CPU-unit-testable — the same reason warm_keep_util.h and session_pool_util.h are.
- Eviction
- LRU on last use, and evicting drops KV ONLY — never message history, which is authoritative and re-prefillable. An evicted session's next turn costs a cold prefill, which is exactly what every turn costs today.
- Version
- 2.12.0
Definition in file session_residency.h.
