Per-sequence state for the gh#98 multi-seq batched decode. More...

#include </home/runner/work/entropic/entropic/src/inference/llama_cpp_backend.h>

Public Attributes
std::unique_ptr< Sampler >	sampler
	Owns the per-request chain.

llama_sampler *	chain = nullptr
	Borrowed native chain (sampled per-idx)

llama_seq_id	seq_id = 0
	KV sequence id.

int	pos = 0
	Next KV position to write.

int	logits_idx = -1
	Batch cell holding current logits.

int	n_gen = 0
	Tokens generated so far.

int	max_tokens = 0
	Per-request generation cap.

bool	active = true
	Still generating?

std::vector< llama_token >	out
	Generated tokens.

std::string	finish = "stop"
	Finish reason.

Detailed Description

Per-sequence state for the gh#98 multi-seq batched decode.

One per request: its own grammar sampler chain (option A), its KV sequence id, the batch cell holding its current logits, and its accumulated output. sampler owns the chain; chain borrows it.

Version: 2.8.0

Definition at line 806 of file llama_cpp_backend.h.

Member Data Documentation

◆ active

bool entropic::LlamaCppBackend::BatchSeq::active = true

Still generating?

Definition at line 814 of file llama_cpp_backend.h.

◆ chain

llama_sampler* entropic::LlamaCppBackend::BatchSeq::chain = nullptr

Borrowed native chain (sampled per-idx)

Definition at line 808 of file llama_cpp_backend.h.

◆ finish

std::string entropic::LlamaCppBackend::BatchSeq::finish = "stop"

Finish reason.

Definition at line 816 of file llama_cpp_backend.h.

◆ logits_idx

int entropic::LlamaCppBackend::BatchSeq::logits_idx = -1

Batch cell holding current logits.

Definition at line 811 of file llama_cpp_backend.h.

◆ max_tokens

int entropic::LlamaCppBackend::BatchSeq::max_tokens = 0

Per-request generation cap.

Definition at line 813 of file llama_cpp_backend.h.

◆ n_gen

int entropic::LlamaCppBackend::BatchSeq::n_gen = 0

Tokens generated so far.

Definition at line 812 of file llama_cpp_backend.h.

◆ out

std::vector<llama_token> entropic::LlamaCppBackend::BatchSeq::out

Generated tokens.

Definition at line 815 of file llama_cpp_backend.h.

◆ pos

int entropic::LlamaCppBackend::BatchSeq::pos = 0

Next KV position to write.

Definition at line 810 of file llama_cpp_backend.h.

◆ sampler

std::unique_ptr<Sampler> entropic::LlamaCppBackend::BatchSeq::sampler

Owns the per-request chain.

Definition at line 807 of file llama_cpp_backend.h.

◆ seq_id

llama_seq_id entropic::LlamaCppBackend::BatchSeq::seq_id = 0

KV sequence id.

Definition at line 809 of file llama_cpp_backend.h.

The documentation for this struct was generated from the following file:

inference/llama_cpp_backend.h

Public Attributes