Skip to content

Hybrid LLM + Symbolic Simulation Architectures

Merged findings from three independent research passes on the state of the art in hybrid symbolic/LLM multi-agent simulation, scoped to Chronicle's three-tier design: a deterministic math tier for all ~1,000 NPCs, a small local-LLM tier for ~30 gossip-hub NPCs, and a large-LLM tier for player conversation.

Findings

  • [DESIGN-INPUT] The three-tier design is exactly where the field has converged; all three reports independently confirm it via HiSim, RumorSphere, and AgentTorch. HiSim (ACL 2024, ~1,000 users, 300 LLM "core" + 700 ABM "ordinary") is the closest scale precedent; RumorSphere (arXiv 2509.02172) scales the same pattern to 1M agents with dynamic core promotion instead of a fixed set; AgentTorch reaches 8.4M agents with only ~100 LLM archetypes and beats a full-LLM configuration on forecasting error. All three reports cite this convergence — report 3 notes Chronicle's ~1,000 NPCs / ~30 hubs sits in a design-space region no published system occupies (game-scale, not social-media-scale), with HiSim as the nearest neighbor.

  • [BUILD-ON] The canonical tier interface is "LLM text → numeric scalar → symbolic update," and it's directly reusable. HiSim and RumorSphere both convert LLM-generated text to a stance/sentiment scalar in [−1, 1] (HiSim: GPT-3.5 stance classification + TextBlob sentiment; RumorSphere: LLM-based "Semantics-to-Scalar" conversion) that feeds a bounded-confidence/DeGroot-style ABM update on the mass tier. All three reports independently identify this as the single most reusable design decision. Report 3 adds the important inversion: symbolic state should be authoritative and LLM text merely an annotated view of it — the LLM never writes raw text into shared state, only scores with provenance.

  • [RISK] "More LLM agents past a small core degrades fidelity" is real but has three distinct mechanisms, not one — and there's a countervailing scaling law for population size. Report 1: RumorSphere/HiSim/MF-LLM show the "average persona" homogenization problem validates capping the LLM tier. Report 3 decomposes this further: a cost claim (RumorSphere's all-LLM ablation costs 10x tokens for no fidelity gain), a diversity claim (MF-LLM: larger backbones like GPT-4o-mini and DeepSeek-R1 underperform Qwen2-1.5B because they produce homogeneous responses), and a calibration claim (AgentTorch: 8.4M-agent archetype config beats a 100-agent full-LLM sub-sample on forecasting error — losing population scale costs more than gaining per-agent LLM reasoning). Countervailing: RumorSphere's 100-agent runs converge to consensus too fast and misalign with real dynamics; the 1M-agent run is realistic. Synthesis (report 3): scale the population as large as the math tier allows; keep the LLM-driven share small and dynamically assigned.

  • [BUILD-ON] Structured-output write-back (constrained JSON decoding) is now a solved, near-zero-overhead engineering problem — all three reports converge on XGrammar as the leading engine. Report 1: XGrammar delivers <40µs/token mask computation, up to 3x speedup on JSON Schema and >100x on CFG vs. prior backends, now integrated in vLLM. Report 2 gives a concrete JSON schema example (dialogue_line, belief_updates array with delta_strength bounded [-1,1], emotional_state_shift enum) and notes 10-50ms compile overhead. Report 3 is most current: JSONSchemaBench shows constrained decoding can speed up generation by up to 50%; XGrammar is now the default in vLLM and SGLang; Guidance/llguidance skips sampling entirely when the grammar uniquely determines the next token (~6-9ms/token vs ~15-16ms unconstrained). All three flag the same caveat: constrained decoding guarantees format, not semantics — schema-valid JSON can still be factually wrong, so a content/provenance validator must sit downstream of the grammar.

  • [BUILD-ON] Local 1B-8B models are practical to co-run with Skyrim on consumer GPUs in 2026, with converging concrete numbers. Report 1: 8B @ Q4_K_M ≈ 90-140 tok/s on RTX 4090, ~42 tok/s on RTX 3060 12GB, ~5GB weights; vLLM continuous batching gives 4-8x (up to 23x) throughput over naive batching. Report 2 gives a VRAM budget table for a 16GB card (game ~8-10GB, 7B FP8 model ~4.8GB weights, KV cache 1.2-2GB) and per-model throughput (Qwen2.5-3B ~180 tok/s, Llama-3.1-8B ~85 tok/s decode). Report 3 is the most detailed and recent: Q4_K_M ≈ 0.56GB per billion params; Skyrim itself uses ~2GB VRAM per Mantella community measurements, so a 4B Q4 model (~3GB + ~1GB KV) fits comfortably alongside it on an 8GB card; at ~50 concurrent requests vLLM's PagedAttention sustains ~920 tok/s aggregate vs. ~155 tok/s for queue-based engines. Recommended models converge across reports: Qwen3-4B/8B or Qwen2.5-3B/7B-Instruct, Llama-3.x 8B, and the Skyrim-specific fine-tune Mantella-Skyrim-Llama-3-8B (report 1).

  • [RISK] Long-running LLM-agent simulations reliably exhibit persona drift, formality collapse, and memory fabrication — all three reports document this with different empirical sources, and it's the most consistently-covered risk across all three. Report 1: Multi-IF shows even o1-preview drops 88%→71% accuracy by turn 3 ("instruction forgetting"); Laban et al. (arXiv 2505.06120, Microsoft/Salesforce, 200K+ simulated conversations) find a 39% average multi-turn performance drop. Report 2 names it "Persona Decay" and "Formality Collapse" with the same context-dilution mechanism. Report 3 adds the counterintuitive finding that larger, more capable models drift more than smaller ones, plus Lifelong-SOTOPIA showing believability declines monotonically across every tested model over multi-episode chains, and quantifies a mitigation: multi-turn RL against persona-consistency rewards reduced inconsistency >55% (NeurIPS 2025). All three agree on the same mitigation direction: never let context-window history alone carry identity — re-inject persona + authoritative symbolic state every turn (ID-RAG per report 1; structured persona cards + lorebook per reports 2/3).

  • [RISK] Memory fabrication / error cascades are a documented, measurable failure mode with security-literature framing. Report 1: GATSim example of an agent reflecting on nonexistent congestion on day 1; "Reflection Repetition Rate" metric. Report 2/3 both cite Project Sid's groundedness failures (agent claims to eat nonexistent food) cascading into dysfunctional behavior. Report 3 is uniquely detailed here: a 2026 claim-level study quantifies hallucination as a stochastic process through agent chains (per-hop attenuation −0.072, p<0.001, but with factual decay), and OWASP has codified this as ASI08 (Cascading Failures) — most pipeline validation is syntactic (valid JSON) not semantic (true claims); industry postmortems put unorchestrated multi-agent production failure rates above 40%. Universal mitigation across all three: the symbolic/numeric tier must be the sole source of truth; the LLM only proposes deltas a validator can reject (report 1 calls this "governance middleware / gated writing"; report 3 calls it "provenance as a hard type constraint" — every belief carries its source event ID, and claims referencing nonexistent events are rejected).

  • [DESIGN-INPUT] Evaluation is the field's acknowledged weak point, not model capability — all three reports state this near-verbatim, citing the same 2025 Springer review. Report 1: "'Validation, not capability, is the bottleneck'" — cites survey-instrument calibration (Stanford 1,000-people study, 0.85 normalized GSS accuracy), regression-scenario replication, and distributional (not point) matching as the credible techniques. Report 3 adds the sharpest caveat: LLM-as-judge is circular — SOTOPIA-π found GPT-4-based evaluation progressively overestimated agents that were tuned against that same judge, so a judge used for training must never also be used for validation. For a game with no real-world ground truth, all three converge on the same practical answer: build regression scenarios with designer-defined target curves (report 1's "seed a rumor, assert propagation/decay"; report 3's DTW/Pearson-against-target-curves) plus a small human-rated believability sample — do not attempt academic-grade survey calibration.

  • [DEFER] Dynamic/adaptive promotion of which NPCs get LLM reasoning each tick (RumorSphere's conflict-boundary criterion) outperforms a static hub list, but its exact mechanism isn't publicly implementable yet. Reports 1 and 3 both flag that RumorSphere's dynamic core selection (promote agents at "information conflict boundaries" via a confusion index τ>0.6) beat both an all-LLM configuration and a static top-degree-core configuration — the static selection actually hurt alignment by letting a fixed elite dominate. But report 1 notes RumorSphere's code and exact CAH update equation / Semantics-to-Scalar prompt are not yet public (promised on acceptance) — reimplementation from the paper is required. Recommendation across reports: start with a static ~30-hub selection by centrality (cheaper, provable now); treat dynamic promotion as a Phase 2 upgrade once static hubs prove too rigid, or once RumorSphere's code ships.

  • [RISK] Motivated reasoning does not emerge for free from LLMs — it must be engineered into the symbolic layer, only mentioned in report 3. Chuang et al. found LLM agents have a default fact-oriented bias that erodes assigned misinformation personas mid-conversation; Taubenfeld et al. found partisan LLM agents fail to polarize in conversation the way humans do (they moderate toward each other instead). Design consequence unique to report 3: the symbolic tier should own the direction of bias (grudge → discount evidence from grudge target, via a numeric credibility multiplier), and the LLM should only own the rationalization text — asking the model to spontaneously generate the bias itself produces "fact-tending mush." This is a direct, actionable constraint on the tier-2 prompt design that the other two reports don't surface.

Details

1. Hybrid architectures and their tier interfaces

HiSim (Mou, Wei & Huang, ACL 2024 Findings; arXiv:2402.16333; github.com/xymou/HiSim). Splits a 1,000-user simulation into 300 LLM-driven "core" users (selected once, statically, by influence ranking) and 700 ABM-driven "ordinary" users (Mesa library; Bounded Confidence / Hegselmann–Krause / Relative Agreement / Social Judgement / Lorenz models). Interface: a shared continuous attitude score a ∈ [−1, 1]; core-agent text is quantized via GPT-3.5 stance classification (support/neutral/oppose) + TextBlob sentiment intensity. Influence is one-directional (LLM core → ABM ordinary only) — HiSim explicitly does not model ordinary→core influence. Micro-evaluation found LLM agents systematically over-commit to clear stances rather than staying neutral, which for a rumor system means the LLM tier will exaggerate conviction unless the numeric tier dampens it.

RumorSphere (arXiv:2509.02172, v2 Jan 2026; builds on HiSim). Scales to 1,000,000 agents, reports a 26.5% average reduction in simulation bias vs. prior baselines (v1 abstract cited 64%, softened to 26.5% in v2). Novelty: a Dynamic Interaction Strategy (DIS) grounded in information-cocoon theory recomputes, each timestep, an information confusion index τᵢ = 2(1 − sᵢ)·dᵢ ∈ [0,1] (sᵢ = opinion similarity to neighbors, dᵢ = neighbor diversity) per agent. Agents with τ > θ (θ=0.6) and sufficient degree are promoted to LLM "core" status for that tick; the rest run Confusion-Adaptive Herding (CAH), a bounded-confidence ABM with opinion score oᵢ ∈ [−1, 1]. Core agents have persona + dual memory (personal/environmental, ranked recency/relevance/importance) + a five-action module (Post, Retweet, Reply, Like, Do Nothing). Removing adaptive grouping (all-LLM) costs 10x tokens for no fidelity gain; static top-degree core selection hurts alignment (premature convergence). Estimated full 1M-agent run cost: ~$48 of GPT-4o-mini tokens. Code and the exact CAH/Semantics-to-Scalar prompt are not yet public — reimplementation from the paper is required.

AgentTorch (arXiv:2409.10568). Compresses the population into archetypes — unique combinations of prompt variables (demographics, disease/policy context) — queries the LLM once per archetype per timestep, converts the response to a Bernoulli behavior probability, and samples the sub-population from that distribution. Differentiable end-to-end (gradient-assisted calibration against historical time series even with LLMs in the loop). Ran at 8.4M agents on commodity hardware; the archetype configuration beat both a pure-heuristic population and a 100-agent full-LLM sub-sample on held-out forecasting error. Trade-off: reduced individual diversity among agents sharing an archetype — acceptable for background NPCs, wrong for named quest characters. Direct transfer to Chronicle: NPCs sharing a (belief-bucket, relationship-to-rumor, trait) tuple can share one tier-2 inference call.

MF-LLM (NeurIPS 2025). Instead of per-agent LLM calls, fine-tunes a single backbone ("IB-Tune") to generate population decision sequences conditioned on a mean-field signal summarizing population state; individuals are sampled from the resulting distribution. Matched a real rumor prevalence checkpoint (predicted 0.645 vs. actual 0.643). Central finding: Qwen2-1.5B-Instruct outperformed GPT-4o-mini and DeepSeek-R1 as backbones, because larger models emit near-identical responses under identical prompts, degrading long-horizon fidelity. Also: low token-level NLL does not imply high rollout fidelity — a model can be locally fluent and globally wrong, arguing against selecting the tier-2 model by perplexity or generic benchmarks.

DeepMind Concordia (github.com/google-deepmind/concordia; v2.0, arXiv:2507.08892, Aug 2025). The most mature general framework and the most directly transplantable pattern for Chronicle's tier-3 conversation layer. A "Game Master" (GM) entity mediates between agent LLM calls and associative memory, structured around typed output requests: MAKE_OBSERVATION, NEXT_ACTION_SPEC (defines the JSON schema the agent must answer in), NEXT_ACTING, RESOLVE (adjudicates an attempted action into a committed outcome), NEXT_GAME_MASTER. State lives in components attached to entities — the LLM proposes, the game master disposes. Apache-licensed Python; supports open-weights models (e.g., Gemma 2 9B via Together AI). Scenario complexity: 4-agent scenarios ~350 lines, 7-agent ~1,000-1,300 lines, complex scenarios 2,000+ lines — budget for component wiring, not just prompt authoring.

AgentSociety / AgentSociety 2 (arXiv:2502.08691; pip install agentsociety2). Flagship all-LLM urban simulator, 10,000+ agents / 5M interactions, validated via survey/interview/intervention experiments (polarization, UBI, hurricanes). AgentSociety 2 adds LLM-native PersonAgents, Ray-based task scaling, and JSONL replay — directly useful as a template for Chronicle's regression-test harness.

SocioVerse (arXiv:2504.10157). Generalizes the LLM-core/ABM-mass lineage into a "world model" over a pool of 10M real user profiles (Social Environment / User Engine / Scenario Engine / Behavior Engine). Validation style — election prediction >90% accuracy, NRMSE + KL divergence for attitude-distribution alignment against real polls — is a useful macro-alignment template for Chronicle's regression scenarios.

Project Sid (Altera, arXiv:2411.00114). Contributes the concurrency pattern needed for real-time play: the PIANO architecture runs ~10 modules (memory, goal generation, social awareness, talking, skill execution, etc.) concurrently as stateless functions over a shared Agent State, with a Cognitive Controller acting as an information bottleneck that synthesizes module outputs into coherent, non-contradictory decisions. Documented failure modes: groundedness failures (agent claims to eat nonexistent food) cascading into dysfunctional behavior, and cross-agent miscommunication (agreed actions diverging from executed ones) — both argue for a Cognitive-Controller-style write bottleneck between Chronicle's rumor-mutation tier and dialogue tier, rather than letting them share a context window.

Vox Deorum (arXiv:2512.18564, Dec 2025). The closest published analog to a shipped hybrid game system: LLM+X inside Civilization V (Vox Populi mod), validated across 2,327 complete games. Exactly Chronicle's tier discipline — the LLM handles macro-strategic reasoning once per turn; all micro-execution stays in the deterministic C++ tactical AI. Interface: game state serialized to a compact structured Markdown document; the LLM's response is a selection from predefined enums, applied as adjustments to "flavor" weight numbers inside the existing symbolic AI — the LLM never touches game state directly, it re-weights a deterministic decision system. Transport chain (Windows named pipe → REST bridge → MCP server exposing getState()/setStrategy()/etc. → LLM client) is a proven template for an SKSE-plugin-to-Python-service architecture. Inference is scheduled to overlap with other players' turns — a latency-hiding pattern directly applicable to background rumor-tier scheduling. Results: GPT-OSS-120B and GLM-4.6 both survived competitively (97% mean survival) with play styles diverging substantially from the symbolic AI and each other — evidence a thin LLM layer over a symbolic core yields behaviorally distinct agents, not noise.

Other systems named: GASim (ACL 2026, arXiv:2605.07692) — graph-based memory retrieval + GAT opinion updates, useful if tier-1 propagation becomes a bottleneck. FDE-LLM (Nature Sci. Reports 2025) — LLM coupled to a dynamical-equation backbone. SAPIENT — Sentinel Layer + orchestrated focus groups, Claude Sonnet 4 / GPT-4o backbone, versioned signal state via structured tool-use. TRUST Agents — atomic claim decomposition to structured logic, trust-weighted voting. OASIS (camel-ai) — event-driven inference manager over async queues, 1M agents, 100% LLM (infra-heavy); reports NRMSE ~30% on propagation curves vs. real X data, and response diversity/helpfulness improving 76.5% scaling 196→10,196 agents.

Framework comparison table (topology / interface / scale), consolidated from report 3:

System Scale LLM share Tier interface style Code
RumorSphere 10⁶ agents ~10² dynamic core Text→score conversion; adaptive grouping Paper only, no official repo
HiSim ~10³ users Static ~300 core Stance+sentiment→ABM; Mesa + AgentVerse github.com/xymou/HiSim
SocioVerse 10⁷ user pool Hierarchical core Four-engine alignment github.com/FudanDISC/SocioVerse (partial)
AgentTorch 8.4×10⁶ agents ~100 archetypes Archetype prompt→Bernoulli sampling; differentiable agenttorch.github.io
MF-LLM population sequences Mean-field conditioned Population signal conditioning + IB-Tune NeurIPS 2025 paper
OASIS 10⁶ agents 100% (infra-heavy) Event-driven inference manager, async queues github.com/camel-ai/oasis
Concordia 2.0 ~10¹ agents 100% GM OutputTypes; entity-component state github.com/google-deepmind/concordia
Project Sid 10–10³ agents 100% PIANO concurrent modules + Cognitive Controller Paper + repo
Vox Deorum 4 players × 375 turns 1 of 4 per turn Markdown state; enum actions→flavor weights arXiv 2512.18564

The four tier-seam patterns, synthesized across all reports:

Seam Direction Canonical implementation Used by
State → prompt rendering Symbolic → LLM Compact structured doc (Markdown/mean-field summary), not raw logs Vox Deorum, MF-LLM, OASIS
Text → numeric scoring LLM → symbolic Stance classifier + sentiment intensity → delta HiSim, RumorSphere
Role gating / promotion Scheduler Dynamic selection of which agents get LLM reasoning each tick RumorSphere DIS, AgentTorch archetypes
Action adjudication LLM ↔ symbolic LLM proposes structured action; symbolic layer validates and commits Concordia RESOLVE, Vox Deorum flavor weights

2. Structured-output write-back patterns

Constrained decoding is the strongest guarantee and is now cheap. vLLM natively supports guided_json/guided_choice/guided_regex/guided_grammar via XGrammar, Outlines, or lm-format-enforcer/llguidance backends.

  • XGrammar (arXiv:2411.15100, mlc-ai/xgrammar): pushdown automaton, <40µs/token mask computation, up to 3x speedup on JSON Schema and >100x on CFG vs. prior backends on Llama-3.1-8B; now the default structured-output backend in both vLLM and SGLang. Independent SLOT study (arXiv:2505.04016) found XGrammar hits 97.1% schema accuracy on complex nested structures (Qwen-2.5-32B) vs. Outlines' 76.4%.
  • Outlines (dottxt-ai/outlines): FSM-based indexing, fast for flat schemas, flattens recursion; grammar compilation can take 3-12s per unique schema (prohibitive if schemas vary per request).
  • Guidance/llguidance (guidance-ai/llguidance): near-zero startup cost, skips sampling entirely when the grammar uniquely determines the next token — ~6-9ms/token vs. ~15-16ms unconstrained; compile time <60ms.
  • JSONSchemaBench (arXiv:2501.10868, 10K real-world schemas, six frameworks): constrained decoding can speed up generation by up to 50% and improves downstream task accuracy by up to 4%, since masking prunes invalid branches of the token space. The 2024-era assumption that constraining costs 5-15% overhead is now obsolete.
  • Caveat (all three reports): format guarantee ≠ semantic guarantee. A grammar-constrained 4B model will emit perfectly parseable nonsense if the underlying judgment is wrong. Schema validation must be paired with content validation (provenance checks, referent-existence checks against the NPC/event roster, confidence bounds) in the symbolic layer.

Function calling / fine-tuned action models. Gigax (github.com/GigaxGames/gigax, MIT license) is purpose-built: fine-tuned NPC models (Llama-3/Phi-3/Mistral bases, GGUF-quantized) take a structured scene description and emit a typed action (e.g., say NPC1 "Hello, Captain!") via an NPCStepper class using Outlines guided generation, sub-1-second GPU inference. On the Berkeley Function-Calling Leaderboard, fine-tuned 7-8B specialists (ToolACE-8B ≈91.5, FunRL-tuned Qwen2.5-Coder-7B 86.0) match or beat GPT-4-0125 (83.4) — a 7-8B function-calling specialist is a legitimate tier-3 option if kept separate from the dialogue model.

Four write-back architecture patterns, ordered most-to-least conservative (report 3): 1. Score sidecar — LLM produces narrative text; a separate extraction step (stance classifier / sentiment scorer / second constrained call) converts it to numeric deltas (HiSim, RumorSphere's seam; cheapest, extractor can be 1B-3B). 2. Dual-channel generation — one constrained call emits both utterance (free text) and state_delta (grammar-constrained JSON) fields in a single schema — the natural tier-3 pattern (Concordia's action specs). 3. Tool/function calling — dialogue model invokes typed functions (record_rumor(subject, predicate, object, confidence, source)). 4. Game-master adjudication — LLM only proposes in natural language; a symbolic resolver interprets and commits (Vox Deorum's flavor-weight adjustment — the safest, most extreme form).

Example belief-update schema (from report 2, illustrative): 

{
  "dialogue_line": { "type": "string" },
  "belief_updates": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "target_npc_id": { "type": "string" },
        "belief_topic": { "type": "string" },
        "delta_strength": { "type": "number", "minimum": -1.0, "maximum": 1.0 },
        "evidence_source": { "type": "string" }
      },
      "required": ["target_npc_id", "belief_topic", "delta_strength", "evidence_source"]
    }
  },
  "emotional_state_shift": { "type": "string", "enum": ["anger", "fear", "trust", "neutral", "contempt"] }
}

Memory write-back reference. The Generative Agents memory stream remains the standard: every event stored as an atomic natural-language observation with timestamp, embedding, and LLM-assigned poignancy; retrieval scores recency × relevance × importance; reflection triggers when cumulative poignancy crosses a threshold, writing synthesized insights back as retrievable thought nodes. Practitioner lesson: store observations as natural-language sentences (embeddings retrieve them better than JSON), and tag reflection writes with provenance type "inference" so they can decay/be contradicted differently from witnessed events.

Small-model reliability expectation. Format compliance is guaranteed by the grammar, but semantic reliability degrades with schema complexity faster than with frontier models — so schemas for 1B-8B writers should be flat, few-fielded, and enum-heavy. Constrain only the metadata fields (mutation type, confidence, target); leave genuinely generative content (mutated rumor text) free-form within a grammar-delimited string field. Watch for reasoning-token blowup in "thinking" variants of small models — run tier-2/tier-3 local models in non-thinking/instruct mode with explicit short-output instructions.

3. Local small-model practicality in 2026

Model shortlist (converges across all three reports): Qwen3-4B-Instruct-2507 / Qwen2.5-3B / Qwen2.5-7B-Instruct (Apache 2.0, strong structured-output and agentic behavior), Llama-3.x 8B / Llama-3.2 1B-3B (most mature tooling, weakest license), Gemma-3/Gemma-2-9B-It, Phi-4-mini (3.8B, MIT), SmolLM3-3B (Apache 2.0, fully open pipeline — best redistribution story for a free mod). Roleplay-tuned derivatives: Stheno 8B (Sao10K), Mistral-Nemo 12B. Skyrim-specific: Mantella-Skyrim-Llama-3-8B (art-from-the-machine, GGUF), fine-tuned on 8,800+ player↔NPC interactions.

Quantization / VRAM (concrete numbers, cross-checked across reports): - Q4_K_M ≈ 4.5 bits/param ≈ 0.56 GB VRAM per billion parameters for weights (report 3's cleanest formula); Q5_K_M ≈0.69 GB/B; Q8_0 ≈1.0 GB/B. - 8B @ Q4_K_M: ~5 GB weights; ~90-140 tok/s on RTX 4090 (~104 tok/s one benchmark), ~52 tok/s RTX 4070, ~42 tok/s RTX 3060 12GB (report 1). Report 3's cross-engine figure: single-stream 8B-class inference on RTX-4090-class cards sits at ~62-71 tok/s once quantization is equalized. - 4B model: should sustain well over 100 tok/s (~3 GB weights + ~1 GB KV cache) — fits an 8GB card alongside Skyrim's own ~2GB VRAM footprint (Mantella community measurement). - 14B @ Q4_K_M: ~69 tok/s (4090), ~23-33 tok/s (3060/4070). - KV cache: ~1.0 GB per active 8K-context sequence at FP16; FP8/paged KV-cache quantization halves this, enabling ~8 parallel background requests in ~1.5 GB. - Rule of thumb: a larger model at Q4 usually beats a smaller model at Q8 in the same memory budget — spend VRAM on parameters, not precision. Q4 vs Q8 is ~1.7x faster at batch size 1 since inference is memory-bandwidth-bound. - CPU fallback: 2-3B models run at 40-60 tok/s on an 8-core laptop CPU — a legitimate degraded accessibility mode for low-end users.

Batching for background inference alongside the game. vLLM continuous batching (iteration-level scheduling + PagedAttention) is the consistent recommendation across all three reports: new requests join the running batch the moment a slot frees, giving 4-8x throughput over naive batching (report 1 cites up to 23x) and, at ~50 concurrent requests, ~920 tok/s aggregate vs. ~155 tok/s for queue-based engines (report 3). Concrete co-residency levers (report 3, most detailed): - Prefix caching — hub persona prompts share long static prefixes (personality card, world rules); removes prefill cost per request; single highest-value optimization for this workload. - Sleep/wake cycling (vLLM sleep mode) — offloads weights to CPU RAM, discards KV cache, frees >90% GPU memory; ~2-3s (L1) or ~7-8s (L2) wake time; can timeshare tier-2 and tier-3 models on one GPU. - Turn-aligned scheduling — trigger mutation batches on game events (cell changes, dialogue endings) rather than a free-running loop, following Vox Deorum's pattern of inferring during other players' turns, so GPU contention never lands mid-frame. - CUDA stream prioritization / chunked prefill — game rendering on a high-priority stream, background LLM on a low-priority compute queue; cap max_num_batched_tokens (e.g., 512) to prevent large prefill phases from spiking frame time.

4. Evaluation methodology

Three convergent layers, from most to least rigorous:

  1. Survey-instrument calibration against real humans (gold standard). Park et al., "Generative Agent Simulations of 1,000 People" (arXiv:2411.10109): agents built from 2-hour interviews replicate participants' GSS answers at 0.85 normalized accuracy (agent accuracy ÷ the human's own two-week test-retest consistency), 0.80 on Big Five, only 0.66 on economic games. Demographics-only agents scored 0.74 vs. 0.86 for interview-grounded agents — quantifying what thin personas cost. The normalization-by-self-consistency trick (don't ask if the sim is "right," ask if it's as consistent as a real person) is the key methodological import, worth borrowing for automated drift detection even without academic-grade calibration.

  2. Regression scenarios / replication of known dynamics. Reproduce established phenomena with known-correct outcomes: rumor tipping points (MF-LLM matched real data at step ~75), echo-chamber formation (HiSim), herd effects (OASIS). For Chronicle: scripted scenarios with invariant assertions on symbolic state (a rumor seeded at NPC A reaches the gossip-hub network within N days; a debunking event decays belief confidence on a designer-set half-life; provenance chains terminate at real events; belief strengths stay in bounds). Deterministic, cheap, CI-friendly — the recommended default for the math tier.

  3. Statistical/distributional matching. HiSim: deviation-from-mean (bias), standard deviation (diversity), Dynamic Time Warping distance, Pearson correlation vs. real attitude curves; stance-prediction accuracy >70%, action-type accuracy >75%, content cosine similarity ~0.7. RumorSphere: ΔBias, ΔDiv, 26.5% opinion-bias reduction. SocioVerse: NRMSE + KL divergence for attitude-distribution alignment vs. real polls. For Chronicle, run the same DTW/distribution-alignment comparisons against designer-defined target curves rather than real-world ground truth.

  4. Expert/human rating and LLM-as-judge — used widely but flagged as unreliable by all sources that discuss it in depth. LLM reward models correlate only ρ≈0.61-0.69 with expert humans and systematically over-score. Report 3's sharpest addition: SOTOPIA-π found LLM-judge evaluation is circular — it progressively overestimated agents that had been tuned against that same judge. Rule: never validate with the judge you trained against. SOTOPIA-EVAL's seven dimensions (goal completion, believability, knowledge acquisition, secret-keeping, relationship, social rules, financial outcomes) with published human-agreement statistics is the most developed micro-instrument, though agreement ranges widely (~0.2-0.95) by dimension.

The meta-finding, stated near-identically in all three reports: a 2025 Springer/AI Review survey concludes validation — not model capability — is the field's central, unsolved challenge; simulations should be judged on distributions of plausible trajectories, not point predictions. Games-specific gap: nothing in the literature addresses save/load persistence over hundreds of real-time hours with a human in the loop — report 3 explicitly names this as the contribution Chronicle itself would make to the field if instrumented and published.

5. Documented failure modes and mitigations

Failure mode Signature Where it hits Chronicle Mitigation
Memory fabrication / confabulation Agents "reflect" on events that never happened (GATSim); confident-but-wrong interpretations persist after environment resets; Reflection Repetition Rate (RRR) metric detects it Tier 2/3 → belief store Numeric tier is sole source of truth; LLM only proposes deltas a schema+logic validator can reject ("governance middleware / gated writing," SSGM arXiv:2603.11768); hard provenance typing — every belief carries a source event ID, claims referencing nonexistent events are rejected
Persona/identity drift, formality collapse Multi-IF: o1-preview drops 88%→71% accuracy by turn 3; Laban et al. (200K+ conversations): 39% average multi-turn performance drop; larger models drift more than smaller ones; Lifelong-SOTOPIA: believability declines monotonically across episodes for every tested model Tier 3 dialogue, tier 2 hub voices ID-RAG (structured identity knowledge-graph retrieved every decision, keeps persona drift ≲0.2%); re-inject persona + authoritative symbolic beliefs every turn; bound context to short window + running summary; reset sessions rather than letting history grow unbounded; multi-turn RL against persona-consistency rewards cut inconsistency >55% (NeurIPS 2025)
Homogenization / "average persona" / Replicant Effect 20-300x less behavioral variance than real humans; MF-LLM: larger backbones produce homogeneous responses; telephone-game studies show iterated generation converges to model-specific stylistic attractors; unprompted LLM partisans moderate toward each other instead of polarizing The mechanism behind "more LLM agents degrades fidelity"; tier-2 mutation, population-level dynamics Keep LLM tier small (~30 hubs validated by literature); small backbones for population-touching tier (MF-LLM); rich, differentiated persona conditioning (interview-grounded > demographic stub, per Stanford 0.74-vs-0.86 gap); higher temperature for diversity where distribution fidelity matters more than any single output; solver-sampler mismatch warning — reasoning-optimized models can be worse at simulating boundedly-rational humans, so don't default to the "smartest" model for the gossip tier
Error propagation / cascades Per-hop hallucination dynamics (2026 claim-level study: −0.072 attenuation per transition but with factual decay); OWASP ASI08 "Cascading Failures" — hallucination/memory poisoning/tool corruption propagate via shared memory and agent-to-agent trust; industry postmortems: unorchestrated multi-agent production failure rates >40% Rumor chains passed hub-to-hub Confidence + mutation-count metadata riding with rumor text between tiers; chain-depth caps or periodic re-grounding against the original event record; a single adjudicating write-bottleneck component (Project Sid's Cognitive Controller pattern) prevents concurrent LLM writers from interleaving contradictory state
Excessive trust / misinformation amplification Generative agents inherit base-model hallucinations, reinforce biases Rumor propagation through hub network Fact-checker agents/filters measurably reduce rumor spread (UT-Austin framework, github.com/UT-SysML/rumors-in-multi-agent); numeric-tier authority pattern contains blast radius
Conformity cascades / groupthink Agents abandon correct beliefs/persona traits under inaccurate majority peer pressure (Asch/Social Impact Theory analogues); BenchForm, Kairos benchmarks quantify it Hub-to-hub gossip network dynamics Isolated private evaluation before public exchange (agent documents private stance from internal state before seeing peer messages); source-weighted trust filtering by historical affinity/credibility
LLM-judge circularity Evaluator systematically overestimates agents tuned against it (SOTOPIA-π) Evaluation harness Held-out human rating; never validate with the same judge used in training/tuning
GPU contention with the running game Frame-time spikes from co-resident background inference Deployment Turn/event-aligned batching, vLLM sleep-mode VRAM cycling, prefix caching, chunked prefill, low-priority CUDA stream for inference

Temporal realism (report 2, unique contribution): standard synchronous/turn-based polling produces artificially uniform (near-Poisson) agent activity timing. A self-excited Hawkes point process as a timing gate, decoupled from the LLM content-generation call, elevates agent activity burstiness to human-like levels without touching prompt structure — the LLM is queried only when the timing gate fires, so timing and "what to say" are separately controlled. This is a technique the other two reports don't cover and is worth evaluating for Chronicle's tier-2 scheduling (when a gossip hub "decides" to spread a rumor vs. simply being available to).