batch_util.h File Reference

gh#98 (v2.8.0) same-prefix batch-generation decision logic. More...

#include <algorithm>
#include <cstddef>
#include <vector>
#include "warm_keep_util.h"

Include dependency graph for batch_util.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces
namespace	entropic
	Activate model on GPU (WARM → ACTIVE).

Functions
template<typename Tok >
std::size_t	entropic::batch_shared_prefix_len (const std::vector< std::vector< Tok > > &seqs)
	Longest shared token prefix across N request sequences (gh#98).
bool	entropic::batch_is_viable (std::size_t n, int n_parallel, std::size_t shared, bool hybrid, std::size_t total_suffix, int n_batch)
	Decide whether the same-prefix batch fast-path is safe + worthwhile.

Detailed Description

gh#98 (v2.8.0) same-prefix batch-generation decision logic.

Pure functions (no llama.cpp, no I/O) so the correctness-critical shared- prefix computation is unit-testable on the CPU pre-commit tier. The backend wraps these with the actual KV mutation (prefill seq 0 → seq_cp fan-out → per-seq suffix decode → batched generation).

Version

2.8.0

Definition in file batch_util.h.