Multi-model lifecycle and routing orchestrator. More...

#include <entropic/inference/orchestrator.h>

Classes
struct	SpeculativeCompatInfo
	Result of a speculative-decoding compatibility check. More...

Public Types
enum class	ResidencyEvent : int { Loaded = 0 , Evicted = 1 , ActivationSwap = 2 }
	Residency observer event codes — mirror the C ABI enum `entropic_residency_event_t` exactly (LOADED=0, EVICTED=1, ACTIVATION_SWAP=2). More...

using	ResidencyObserverFn = std::function< void(ResidencyEvent event, const std::string &tier_name, const std::string &model_path, size_t footprint)>
	Residency observer callback type (internal C++ form).

Public Member Functions
bool	initialize (const ParsedConfig &config)
	Initialize from parsed config.

void	shutdown ()
	Shutdown — unload all models.

	~ModelOrchestrator ()
	Destructor — invokes shutdown() and AdapterManager::unload_all().

GenerationResult	generate (const std::vector< Message > &messages, const GenerationParams &params, const std::string &tier_name="")
	Generate using routed or explicit tier.

GenerationResult	generate (const std::vector< Message > &messages, const GenerationParams &params, std::atomic< bool > &cancel, const std::string &tier_name="")
	Batch generation with cancel support.

std::vector< GenerationResult >	generate_batch (const std::vector< std::vector< Message > > &messages_list, const std::vector< GenerationParams > &params_list, const std::vector< std::string > &tiers, std::atomic< bool > &cancel)
	Same-prefix batch generation on a shared resident model (gh#98).

GenerationResult	generate_streaming (const std::vector< Message > &messages, const GenerationParams &params, std::function< void(std::string_view)> on_token, std::atomic< bool > &cancel, const std::string &tier_name="")
	Streaming generation.

std::string	route (const std::vector< Message > &messages)
	Route to tier using router model.

RoutingResult	last_routing_result () const
	Last routing result.

std::string	last_used_tier () const
	Last used tier name.

std::vector< std::string >	loaded_models () const
	Currently loaded model tier names.

std::vector< std::string >	available_models () const
	All configured tier names.

bool	can_handoff (const std::string &from, const std::string &to) const
	Check if handoff is permitted.

ChatAdapter *	get_adapter (const std::string &tier_name) const
	Get adapter for a tier.

InferenceBackend *	get_backend (const std::string &tier_name) const
	Get the inference backend for a tier (for evaluation APIs).

InferenceBackend *	ensure_model (const std::string &tier_name)
	Get a tier's backend, loading it if it is not resident (gh#157).

entropic_error_t	release_models (const std::string &tier_name)
	Release resident model(s), keeping every registration (gh#164).

bool	tier_declares_vision (const std::string &tier_name) const
	Whether a tier is configured for vision, per CONFIG (gh#157).

AdapterManager &	adapter_manager ()
	Access the LoRA adapter manager.

GrammarRegistry &	grammar_registry ()
	Access the grammar registry.

std::vector< GenerationRecord >	generation_records () const
	Per-generation metric records, oldest first (gh#154).

ProfileRegistry &	profile_registry ()
	Access the GPU resource profile registry.

ThroughputTracker &	throughput_tracker ()
	Access the throughput tracker.

size_t	load_grammars_from (const std::filesystem::path &grammar_dir)
	Load grammars from an explicit directory path.

void	clear_all_prompt_caches ()
	Invalidate prompt/KV caches across every pooled backend.

bool	has_vision_capable_tier () const
	Return true if any configured tier declares the `"vision"` capability (gh#41, v2.1.8).

SpeculativeCompatInfo	check_speculative_compat () const
	Check whether the currently-configured target/draft pair is compatible for speculative decoding.

void	set_speculative_enabled (bool enabled)
	Runtime toggle for the speculative-decoding path.

void	set_residency_observer (ResidencyObserverFn cb)
	Register a residency observer.

std::string	residency_snapshot_json () const
	Serialize the current residency set as a JSON string.

void	forget_session_kv (const std::string &session_key)
	Drop one session's resident KV on every loaded tier (gh#165).

size_t	vram_budget_bytes () const
	Engine-tracked VRAM budget in bytes (0 = unknown).

size_t	tier_footprint_bytes (const std::string &tier_name) const
	Estimated VRAM footprint for a given tier in bytes.

entropic_error_t	last_residency_error () const
	Last residency-related error code, or `ENTROPIC_OK` if none.

void	clear_last_residency_error ()
	Clear `last_residency_error()`.

std::string	select_vision_tier () const
	Pick the canonical vision-capable tier name (gh#41).

void	apply_tier_sampler_defaults_for_test (GenerationParams &params, const std::string &tier_name)
	Test-only forwarder to the private per-tier sampler default application (gh#94, audit task #71).

Static Public Attributes
static constexpr size_t	kMaxGenerationRecords = 64
	Ring capacity for generation_records().

Detailed Description

Multi-model lifecycle and routing orchestrator.

Manages model pool deduplication, per-tier adapters, VRAM lifecycle, and digit-based tier routing via a router model.

Version: 1.8.2

Definition at line 118 of file orchestrator.h.

Member Typedef Documentation

◆ ResidencyObserverFn

using entropic::ModelOrchestrator::ResidencyObserverFn = std::function<void( ResidencyEvent event, const std::string& tier_name, const std::string& model_path, size_t footprint)>

Residency observer callback type (internal C++ form).

Fires synchronously from inside get_model / deactivate_current_if_needed while the swap mutex is held. Must not call back into the orchestrator on the same thread.

Parameters

event	Lifecycle event code.
tier_name	Tier whose backing model changed VRAM state.
model_path	Resolved GGUF path for the model.
footprint	Estimated VRAM footprint in bytes. @callback

Version: 2.2.4

Definition at line 511 of file orchestrator.h.

Member Enumeration Documentation

◆ ResidencyEvent

enum class entropic::ModelOrchestrator::ResidencyEvent : int

strong

Residency observer event codes — mirror the C ABI enum entropic_residency_event_t exactly (LOADED=0, EVICTED=1, ACTIVATION_SWAP=2).

Version: 2.2.4

Definition at line 491 of file orchestrator.h.

Constructor & Destructor Documentation

◆ ~ModelOrchestrator()

entropic::ModelOrchestrator::~ModelOrchestrator ( )

Destructor — invokes shutdown() and AdapterManager::unload_all().

Orchestrate teardown order (gh#58 close-out).

Combines two fixes:

gh#63 (v2.2.9): make shutdown() actually run on destroy, so VRAM release on handle teardown is explicit rather than relying on the shared_ptr<LlamaCppBackend> cascade. Pre-v2.2.9 nothing called shutdown() during the destroy path.
gh#58 close-out (v2.3.0): also call AdapterManager::unload_all() so loaded LoRA llama_adapter_lora* handles don't leak. Pre- v2.3.0 AdapterManager had no destructor — same shape as the pre-v2.2.8 LlamaCppBackend leak.

Teardown order matters: backends first (frees llama_contexts), then adapter handles (safe because the contexts that referenced HOT adapters are gone). Out-of-line so the .cpp can stage the two calls in the right sequence and keep the header free of implementation noise.

@utility

Version: 2.3.0

See header.

Backends first (frees the llama_contexts), LoRA handles second — a LoRA handle must outlive every context that referenced it, so the reverse order is a use-after-free.

@req REQ-INFER-002

Version: 2.3.0

Definition at line 338 of file orchestrator.cpp.

Member Function Documentation

◆ adapter_manager()

AdapterManager & entropic::ModelOrchestrator::adapter_manager ( )

inline

Access the LoRA adapter manager.

Returns: Reference to AdapterManager. @utility

Version: 1.9.2

Definition at line 358 of file orchestrator.h.

◆ apply_tier_sampler_defaults_for_test()

void entropic::ModelOrchestrator::apply_tier_sampler_defaults_for_test	(	GenerationParams &	params,
		const std::string &	tier_name
	)

inline

Test-only forwarder to the private per-tier sampler default application (gh#94, audit task #71).

Exposes ModelOrchestrator::apply_tier_sampler_defaults so a CPU unit test can assert the effective sampler values produced by the MEMBER path — specifically the tier.X -> ov.X 9-line hand-copy (orchestrator.cpp) that the free apply_tier_sampler_overrides tests cannot reach. The tier is looked up in config_, so the caller must have a config assigned (initialize() assigns config_ before any model load, so a failed init still populates it). Pure forwarder — no behavior of its own.

Parameters

params	Generation params (mutated in place).
tier_name	Tier whose frontmatter sampler config to apply. @dg_internal

Version: 2.8.0

Definition at line 642 of file orchestrator.h.

◆ available_models()

std::vector< std::string > entropic::ModelOrchestrator::available_models ( ) const

All configured tier names.

Version: 1.8.2

@dg_internal

Version: 2.1.11

Definition at line 1898 of file orchestrator.cpp.

◆ can_handoff()

bool entropic::ModelOrchestrator::can_handoff	(	const std::string &	from,
		const std::string &	to
	)		const

Check if handoff is permitted.

Version: 1.8.2

Parameters

from	Source tier name.
to	Candidate destination tier name.

Returns: true only when from has an explicit rule set listing to; an empty rule set denies every pair. @req REQ-INFER-020

Version: 1.8.2

Definition at line 2053 of file orchestrator.cpp.

◆ check_speculative_compat()

ModelOrchestrator::SpeculativeCompatInfo entropic::ModelOrchestrator::check_speculative_compat ( ) const

Check whether the currently-configured target/draft pair is compatible for speculative decoding.

Speculative compatibility check (target vs draft).

Reads the active main tier as the target (verifier) and the "draft" role on the SecondaryModelLoader as the proposer. Returns compatible=false with a specific diagnostic when:

No main tier is loaded (target unavailable).
No draft role is loaded (no proposer configured).
entropic::speculative::check_compat rejects the pairing (recurrent target, tokenizer mismatch, etc.).

Returns: SpeculativeCompatInfo. @utility

Version: 2.1.11

Reads the active main tier as the target and the "draft" slot on the secondary loader as the draft. Returns a structured diagnostic the C ABI can forward to consumers.

Returns: SpeculativeCompatInfo with compatible flag + diagnostic. @dg_internal

Version: 2.1.11

Definition at line 2357 of file orchestrator.cpp.

◆ clear_all_prompt_caches()

void entropic::ModelOrchestrator::clear_all_prompt_caches ( )

Invalidate prompt/KV caches across every pooled backend.

Invalidate prompt caches across every pooled backend.

Used when identity content (system prompt prefix) changes so no cached prefix is served against the new prompt. (P1-7, 2.0.6-rc16)

@utility

Version: 2.0.6-rc16

Called on identity content changes so no cached prefix is served against the new system prompt. (P1-7, 2.0.6-rc16). Fans out to secondary roles (router, draft) via SecondaryModelLoader (v2.1.11).

@req REQ-INFER-020

Version: 2.1.11

Definition at line 2248 of file orchestrator.cpp.

◆ clear_last_residency_error()

void entropic::ModelOrchestrator::clear_last_residency_error ( )

inline

Clear last_residency_error().

@utility

Version: 2.2.4

Definition at line 605 of file orchestrator.h.

◆ ensure_model()

InferenceBackend * entropic::ModelOrchestrator::ensure_model ( const std::string & tier_name )

Get a tier's backend, loading it if it is not resident (gh#157).

Public residency-gated activation (gh#157).

The public face of the residency-gated activation path the generation entry points already use: admits the tier against the VRAM budget, swaps out the incumbent, activates, fires ResidencyEvent::Loaded and preloads the model's LoRA adapters.

This exists because models.defer_load makes "no model is loaded yet" a normal state rather than a startup transient, and the non-generation consumers of a backend — entropic_state_save, entropic_state_load, entropic_get_logprobs, entropic_compute_perplexity, entropic_adapter_load — used to reach straight for get_backend and throw "model not active". They lazy-load through here instead.

Takes the swap mutex. Callers that hold it must use get_model.

Parameters

tier_name Tier to make resident.

Returns: ACTIVE backend, or nullptr when the tier is unknown, refused by the residency gate, or fails to load. @utility @req REQ-INFER-019

Version: 2.13.0

See header.

Parameters

tier_name Tier to make resident.

Returns: ACTIVE backend, or nullptr. @utility @req REQ-INFER-019

Version: 2.13.0

Definition at line 1931 of file orchestrator.cpp.

◆ forget_session_kv()

void entropic::ModelOrchestrator::forget_session_kv ( const std::string & session_key )

Drop one session's resident KV on every loaded tier (gh#165).

Called after entropic_session_context_set replaces a conversation, so the restored history cannot decode against the prefix the conversation it replaced left in the cache.

Takes generation_mutex_ — it touches the live llama_context, which a concurrent decode is using. The resulting order from the facade is api_mutex -> generation_mutex_ -> swap_mutex_, and it does not close a cycle: gh#109 removed api_mutex from every run entry point, so nothing holds it while taking generation_mutex_ from the other side.

Parameters

session_key Session whose KV to drop; "" is the default. @req REQ-LOOP-010

Version: 2.13.0

Tiers may share a backend, so the same pointer is visited once.

Parameters

session_key Session whose KV to drop. @req REQ-LOOP-010

Version: 2.13.0

Definition at line 2951 of file orchestrator.cpp.

◆ generate() [1/2]

GenerationResult entropic::ModelOrchestrator::generate	(	const std::vector< Message > &	messages,
		const GenerationParams &	params,
		const std::string &	tier_name = `""`
	)

Generate using routed or explicit tier.

Generate response using routed or explicit tier.

Parameters

messages	Conversation history.
params	Generation parameters.
tier_name	Explicit tier, or empty for routing.

Returns: GenerationResult with routing/swap timing.

Version: 1.8.2

Speculative routing added in v2.1.11 (gh#36): when the kernel is configured and the target/draft pair is compatible, dispatches through LlamaCppBackend::generate_speculative_with_draft; falls back to plain decode otherwise. The dispatch decision is delegated to run_generate_dispatch to keep this method under the SLOC gate. v2.2.4 (gh#57): a refused activation now reports ENTROPIC_ERROR_TIER_MODEL_TOO_LARGE via build_no_model_error instead of the generic GENERATE_FAILED.

gh#154 (v2.13.0): the tier's grammar is resolved BEFORE the model is, so a tier naming a grammar nobody registered returns ENTROPIC_ERROR_GRAMMAR_NOT_FOUND without a swap, a prefill or a token.

Parameters

messages	Conversation history.
params	Generation parameters.
tier_name	Explicit tier or empty for routing.

Returns: GenerationResult. @dg_internal

Version: 2.13.0

Definition at line 808 of file orchestrator.cpp.

◆ generate() [2/2]

GenerationResult entropic::ModelOrchestrator::generate	(	const std::vector< Message > &	messages,
		const GenerationParams &	params,
		std::atomic< bool > &	cancel,
		const std::string &	tier_name = `""`
	)

Batch generation with cancel support.

Batch generate with cancel — see header for contract.

(gh#81, v2.4.2)

Mirrors generate plus a cancel-flag reference; the backend polls the flag per decode step. Closes the gh#81 60s-lag gap where the batch path had no path to honor mid-decode cancel.

Version: 2.4.2

Bypasses run_generate_dispatch (speculative routing) because speculative kernels live on the streaming path; batch only ever calls plain decode. Calls model->generate(messages, params, cancel) which polls cancel per token.

gh#154 (v2.13.0): carries the same pre-model tier-grammar gate as the other overload — one rule, every entry point.

@dg_internal

Version: 2.13.0

Definition at line 866 of file orchestrator.cpp.

◆ generate_batch()

std::vector< GenerationResult > entropic::ModelOrchestrator::generate_batch	(	const std::vector< std::vector< Message > > &	messages_list,
		const std::vector< GenerationParams > &	params_list,
		const std::vector< std::string > &	tiers,
		std::atomic< bool > &	cancel
	)

Same-prefix batch generation on a shared resident model (gh#98).

Same-prefix batch generation on a shared model — see header (gh#98).

All N requests run on ONE backend (resolved from the lead tier — they share the model pool). Each request's params are resolved per its tier (grammar + samplers), then dispatched to the backend's generate_batch, which prefills the shared prompt prefix once and fans out. Adapter parse is applied per result.

Parameters

messages_list	Per-request message lists (built with each tier's system prompt so same-tier requests share a prefix).
params_list	Per-request base params.
tiers	Per-request tier names ("" = lead/default).
cancel	Cancel flag.

Returns: One GenerationResult per request, in input order.

Version: 2.8.0

All requests resolve to ONE backend (the lead tier's model — tiers share the pool). Each request's params are resolved per its own tier (grammar + samplers), then the backend's generate_batch prefills the shared prefix once and fans out. Tool staging is per-model, so this path targets grammar-constrained requests (params.grammar), not common_chat tool injection.

gh#154 (v2.13.0): every arm's grammar resolves before the shared model is touched, and one arm naming an unregistered grammar refuses the whole batch — a single decode over a shared prefill cannot run half-constrained.

@dg_internal

Version: 2.13.0

Definition at line 948 of file orchestrator.cpp.

◆ generate_streaming()

GenerationResult entropic::ModelOrchestrator::generate_streaming	(	const std::vector< Message > &	messages,
		const GenerationParams &	params,
		std::function< void(std::string_view)>	on_token,
		std::atomic< bool > &	cancel,
		const std::string &	tier_name = `""`
	)

Streaming generation.

Streaming generation with speculative dispatch.

Version: 1.8.2

Speculative routing added in v2.1.11 (gh#36): when the kernel is configured and the target/draft pair is compatible, dispatches to LlamaCppBackend::generate_speculative_with_draft via try_speculative_route_streaming. Falls back to plain streaming on NOT_SUPPORTED or compatibility failure, with a diagnostic logged.

gh#108 (v2.10.0): on_token is wrapped with StreamThinkFilter so thinking-channel tokens are stripped from the live stream. The buffered result.content is post-processed via apply_adapter_parse on return, mirroring the non-streaming generate() path. This also enables MTP streaming (the streaming guard in mtp_unsupported_reason is removed in the same gh#108 v2.10.0 change).

gh#108 (v2.10.3): the filter's markers come from the resolved adapter, the same source the buffered strip uses — v2.10.0 left it on a hardcoded <think> pair that gemma4 never emits.

gh#154 (v2.13.0): the tier-grammar gate runs before get_model, so an unregistered tier grammar refuses the stream without emitting a token.

Parameters

messages	Conversation history.
params	Generation parameters.
on_token	Per-token callback, wrapped by the reasoning filter.
cancel	Cancel flag, polled by the backend once per token.
tier_name	Explicit tier, or empty to route.

Returns: GenerationResult with content parsed by the shared rule; an ENTROPIC_ERROR_GENERATE_FAILED result when no model resolves for the tier, or ENTROPIC_ERROR_GRAMMAR_NOT_FOUND when its grammar is unregistered. @req REQ-INFER-011 @req REQ-INFER-005 @req REQ-INFER-008 @req REQ-INFER-007

Version: 2.13.0

Definition at line 1067 of file orchestrator.cpp.

◆ generation_records()

std::vector< GenerationRecord > entropic::ModelOrchestrator::generation_records ( ) const

Per-generation metric records, oldest first (gh#154).

A bounded ring — the last kMaxGenerationRecords generations this orchestrator ran. Serialized into entropic_metrics_json as generations[], which is the only route by which any GenerationResult field has ever reached a C consumer: tok/s, prefill_tokens, the speculative draft/accept counts and the grammar provenance were all computed, logged and then dropped.

Returns: A copy, so the caller never holds the lock.

Version: 2.13.0

Returns: A copy of the ring, so the caller never holds the lock. @req REQ-INFER-008

Version: 2.13.0

Definition at line 2436 of file orchestrator.cpp.

◆ get_adapter()

ChatAdapter * entropic::ModelOrchestrator::get_adapter ( const std::string & tier_name ) const

Get adapter for a tier.

Version: 1.8.2

@dg_internal

Version: 1.8.2

Definition at line 2068 of file orchestrator.cpp.

◆ get_backend()

InferenceBackend * entropic::ModelOrchestrator::get_backend ( const std::string & tier_name ) const

Get the inference backend for a tier (for evaluation APIs).

Get the inference backend for a tier.

Parameters

tier_name Tier name (e.g. "lead", "eng").

Returns: Backend pointer, or nullptr if tier not found.

Version: 1.10.2

Parameters

tier_name Tier name.

Returns: Backend pointer, or nullptr if not found. @utility

Version: 1.10.2

Definition at line 1916 of file orchestrator.cpp.

◆ grammar_registry()

GrammarRegistry & entropic::ModelOrchestrator::grammar_registry ( )

inline

Access the grammar registry.

Returns: Reference to GrammarRegistry. @utility

Version: 1.9.3

Definition at line 366 of file orchestrator.h.

◆ has_vision_capable_tier()

bool entropic::ModelOrchestrator::has_vision_capable_tier ( ) const

Return true if any configured tier declares the "vision" capability (gh#41, v2.1.8).

Vision-capability lookup (gh#41, v2.1.8).

Read-only lookup over the parsed ModelsConfig — does not touch backend state. Used by the facade's entropic_run_messages entry point to short-circuit with ENTROPIC_ERROR_NO_VISION_TIER before dispatching a multimodal turn that no tier can handle.

@utility

Version: 2.1.8

Returns: true if any configured tier declares "vision"; false lets the facade short-circuit with ENTROPIC_ERROR_NO_VISION_TIER instead of dispatching a turn no tier can handle. @req REQ-INFER-025

Version: 2.1.8

Definition at line 2265 of file orchestrator.cpp.

◆ initialize()

bool entropic::ModelOrchestrator::initialize ( const ParsedConfig & config )

Initialize from parsed config.

Initialize orchestrator: backends, routing, adapters, grammars.

Parameters

config Full engine config.

Returns: true on success.

Version: 1.8.2

Adds speculative-draft activation alongside router activation in v2.1.11 (gh#36) — the draft slot loads when inference.speculative. enabled is true and a draft_model is configured. v2.2.4 (gh#57) caches the VRAM budget from ENTROPIC_VRAM_BUDGET_BYTES so the residency gate in get_model has a number to test against.

Parameters

config Parsed engine config.

Returns: true on success. @utility

Version: 2.14.0

Definition at line 268 of file orchestrator.cpp.

◆ last_residency_error()

entropic_error_t entropic::ModelOrchestrator::last_residency_error ( ) const

inline

Last residency-related error code, or ENTROPIC_OK if none.

Set by get_model when a tier-fit check fails (returns ENTROPIC_ERROR_TIER_MODEL_TOO_LARGE). The facade clears it after translating to the C ABI return code. Independent of the last_error_ string carried on individual backends.

@utility

Version: 2.2.4

Definition at line 598 of file orchestrator.h.

◆ last_routing_result()

RoutingResult entropic::ModelOrchestrator::last_routing_result ( ) const

Last routing result.

Version: 1.8.2

Returns: The RoutingResult recorded by the most recent route() — selected tier, previous tier, raw router output, swap action and timings. Default-constructed before the first route. @req REQ-INFER-020

Version: 1.8.2

Definition at line 1855 of file orchestrator.cpp.

◆ last_used_tier()

std::string entropic::ModelOrchestrator::last_used_tier ( ) const

Last used tier name.

Version: 1.8.2

@dg_internal

Version: 1.8.2

Definition at line 1864 of file orchestrator.cpp.

◆ load_grammars_from()

size_t entropic::ModelOrchestrator::load_grammars_from ( const std::filesystem::path & grammar_dir )

Load grammars from an explicit directory path.

Parameters

grammar_dir Path containing .gbnf files.

Returns: Number of grammars loaded.

Version: 2.0.6

Called by the facade after data-dir resolution. This is the fallback path when config_dir doesn't contain a grammars subdir (e.g., installed layout where grammars live under share/entropic).

Parameters

grammar_dir Path to directory containing .gbnf files.

Returns: Number of grammars loaded. @dg_internal

Version: 2.0.6

Definition at line 2227 of file orchestrator.cpp.

◆ loaded_models()

std::vector< std::string > entropic::ModelOrchestrator::loaded_models ( ) const

Currently loaded model tier names.

Version: 1.8.2

Includes "router" when the secondary loader reports the role as loaded (v2.1.11, gh#27 — previously checked the raw router_ field).

Returns: Tier names whose backend reports is_loaded(), plus "router" when the secondary loader holds that role; a role whose load failed is absent. @req REQ-INFER-020

Version: 2.1.11

Definition at line 1880 of file orchestrator.cpp.

◆ profile_registry()

ProfileRegistry & entropic::ModelOrchestrator::profile_registry ( )

inline

Access the GPU resource profile registry.

Returns: Reference to ProfileRegistry. @utility

Version: 2.0.0

Definition at line 393 of file orchestrator.h.

◆ release_models()

entropic_error_t entropic::ModelOrchestrator::release_models ( const std::string & tier_name )

Release resident model(s), keeping every registration (gh#164).

Release resident model(s) (gh#164).

Unloads the backend(s), fires ResidencyEvent::Evicted for every tier that was backed by one, and clears the active-tier record so the next get_model / ensure_model reloads through the residency gate.

What is DELIBERATELY kept: the tier configs, the grammar registry, the footprint estimates, and the LoRA adapter REGISTRATIONS. The swap path (deactivate_current_if_needed) calls unload_all_for_model, which ERASES adapter entries — correct for a tier that is being replaced, wrong for one the consumer intends to bring back. Release frees the llama adapter handles and leaves the registrations COLD, to be re-bound by preload_adapters_for_model at the next activation.

Takes the swap mutex. It does NOT serialize generation — the caller (the facade) holds the handle's turn claim, which is what makes unloading a live context safe.

Parameters

tier_name Tier to release, or empty for every resident model including the secondary roles (router, draft). An MTP head is owned by its target backend and is torn down with it.

Returns: ENTROPIC_OK, or ENTROPIC_ERROR_MODEL_NOT_FOUND for an unknown tier name. Releasing an already-unloaded tier is a no-op. @utility @req REQ-INFER-019

Version: 2.13.0

See header.

Parameters

tier_name Tier to release, or empty for all.

Returns: ENTROPIC_OK, or MODEL_NOT_FOUND for an unknown tier. @utility @req REQ-INFER-019

Version: 2.13.0

Definition at line 1944 of file orchestrator.cpp.

◆ residency_snapshot_json()

std::string entropic::ModelOrchestrator::residency_snapshot_json ( ) const

Serialize the current residency set as a JSON string.

Serialize the current VRAM residency snapshot to JSON.

Schema is documented on the C ABI entry point entropic_residency_snapshot (entropic.h). Read-only — takes the swap mutex briefly to obtain a consistent snapshot.

Returns: JSON object string. @utility

Version: 2.2.4

Returns: JSON object describing budget, tiers and footprints. @dg_internal

Version: 2.14.0

Definition at line 2971 of file orchestrator.cpp.

◆ route()

std::string entropic::ModelOrchestrator::route ( const std::vector< Message > & messages )

Route to tier using router model.

Route to appropriate tier using router model.

Parameters

messages Current conversation.

Returns: Selected tier name.

Version: 1.8.2

Guard updated in v2.1.11: routing requires models.router to be configured (was: router_ non-null). The slot is owned by secondary_loader_ since gh#27.

Every decision is recorded in last_routing_result_ — selected tier, previous tier, raw model output, swap action — and pushed onto a tier history bounded at 5 entries.

Parameters

messages Current conversation.

Returns: Selected tier name; the default tier when routing is disabled, no router is configured, or classification found no mapped digit. @req REQ-INFER-020

Version: 2.1.11

Definition at line 1157 of file orchestrator.cpp.

◆ select_vision_tier()

std::string entropic::ModelOrchestrator::select_vision_tier ( ) const

Pick the canonical vision-capable tier name (gh#41).

First vision-capable tier name (gh#41, v2.1.8).

Returns the first tier (iteration order of the parsed models.tiers map) whose capabilities include "vision", or empty string if none exists. Multi-tier policy refinements (e.g. prefer the default tier when it qualifies) can layer on top later — single-vision-tier deployments are the common case for v2.1.8 (gh#42 ships the primary tier as the only vision-capable bundled entry).

Returns: Vision tier name, or "" if none configured. @utility

Version: 2.1.8

Returns: The canonical vision tier's name, or "" when no configured tier declares the capability. @req REQ-INFER-025

Version: 2.1.8

Definition at line 2279 of file orchestrator.cpp.

◆ set_residency_observer()

void entropic::ModelOrchestrator::set_residency_observer ( ResidencyObserverFn cb )

Register a residency observer.

Register / replace / clear the residency observer.

Replaces the previous one.

Passing an empty std::function clears the observer.

Parameters

cb	Observer callable, or empty to clear. @utility

Version: 2.2.4

@dg_internal

Version: 2.2.4

Definition at line 2876 of file orchestrator.cpp.

◆ set_speculative_enabled()

void entropic::ModelOrchestrator::set_speculative_enabled ( bool enabled )

inline

Runtime toggle for the speculative-decoding path.

Lets consumers (and tests) flip speculative on/off without reinitializing the orchestrator. Defaults to whatever inference.speculative.enabled was in the parsed config at init time.

Parameters

enabled true to route through the speculative kernel when a compatible draft is loaded. @utility

Version: 2.1.11

Definition at line 479 of file orchestrator.h.

◆ shutdown()

void entropic::ModelOrchestrator::shutdown ( )

Shutdown — unload all models.

Version: 1.8.2

Main-tier pool is unloaded directly; secondary roles (router, draft, etc.) are released through secondary_loader_.shutdown() (v2.1.11).

@req REQ-INFER-002 @req REQ-INFER-020

Version: 2.1.11

Definition at line 316 of file orchestrator.cpp.

◆ throughput_tracker()

ThroughputTracker & entropic::ModelOrchestrator::throughput_tracker ( )

inline

Access the throughput tracker.

Returns: Reference to ThroughputTracker. @utility

Version: 2.0.0

Definition at line 401 of file orchestrator.h.

◆ tier_declares_vision()

bool entropic::ModelOrchestrator::tier_declares_vision ( const std::string & tier_name ) const

Whether a tier is configured for vision, per CONFIG (gh#157).

Config-only vision capability for a tier (gh#157).

True when the tier declares the "vision" capability or carries an mmproj path. Deliberately does NOT consult the backend: the backend's answer comes from has_vision_, which is set while the mmproj context is built during ACTIVATION, so an unloaded tier reported "no vision" — a wrong answer, not an unknown one, and models.defer_load makes unloaded the normal state.

Parameters

tier_name Tier name.

Returns: true when the configured tier is vision-capable. @utility @req REQ-INFER-025

Version: 2.13.0

See header.

Parameters

tier_name Tier name.

Returns: true when the tier declares "vision" or carries an mmproj path. @utility @req REQ-INFER-025

Version: 2.13.0

Definition at line 2036 of file orchestrator.cpp.

◆ tier_footprint_bytes()

size_t entropic::ModelOrchestrator::tier_footprint_bytes ( const std::string & tier_name ) const

Estimated VRAM footprint for a given tier in bytes.

Public footprint accessor — memoizes via tier_footprint_bytes_.

Sum of GGUF weights file size and a context-length-derived KV cache estimate plus the configured vram_reserve_mb headroom. Returns 0 if the tier is unknown.

Parameters

tier_name Tier name. @utility

Version: 2.2.4

@dg_internal

Version: 2.2.4

Definition at line 2859 of file orchestrator.cpp.

◆ vram_budget_bytes()

size_t entropic::ModelOrchestrator::vram_budget_bytes ( ) const

inline

Engine-tracked VRAM budget in bytes (0 = unknown).

Sources, in priority order: ENTROPIC_VRAM_BUDGET_BYTES environment override → CUDA cudaMemGetInfo (when the CUDA inference backend is the active build) → 0. When 0, the orchestrator does not enforce a per-tier budget gate.

@utility

Version: 2.2.4

Definition at line 572 of file orchestrator.h.

Member Data Documentation

◆ kMaxGenerationRecords

constexpr size_t entropic::ModelOrchestrator::kMaxGenerationRecords = 64

staticconstexpr

Ring capacity for generation_records().

Version: 2.13.0

Definition at line 385 of file orchestrator.h.

The documentation for this class was generated from the following files:

entropic/inference/orchestrator.h
inference/orchestrator.cpp

Classes

Public Types

Public Member Functions

Static Public Attributes

Detailed Description

Member Typedef Documentation

◆ ResidencyObserverFn

Member Enumeration Documentation

◆ ResidencyEvent

Constructor & Destructor Documentation

◆ ~ModelOrchestrator()

Member Function Documentation

◆ adapter_manager()

◆ apply_tier_sampler_defaults_for_test()

◆ available_models()

◆ can_handoff()

◆ check_speculative_compat()

◆ clear_all_prompt_caches()

◆ clear_last_residency_error()

◆ ensure_model()

◆ forget_session_kv()

◆ generate() [1/2]

◆ generate() [2/2]

◆ generate_batch()

◆ generate_streaming()

◆ generation_records()

◆ get_adapter()

◆ get_backend()

◆ grammar_registry()

◆ has_vision_capable_tier()

◆ initialize()

◆ last_residency_error()

◆ last_routing_result()

◆ last_used_tier()

◆ load_grammars_from()

◆ loaded_models()

◆ profile_registry()

◆ release_models()

◆ residency_snapshot_json()

◆ route()

◆ select_vision_tier()

◆ set_residency_observer()

◆ set_speculative_enabled()

◆ shutdown()

◆ throughput_tracker()

◆ tier_declares_vision()

◆ tier_footprint_bytes()

◆ vram_budget_bytes()

Member Data Documentation

◆ kMaxGenerationRecords