A beautiful world can still feel scripted
The city has burned, but the shopkeeper repeats the same three lines. A companion witnessed last night’s betrayal, yet greets you as if nothing happened. Games can make a world look alive while its inhabitants remain caught in a script.
Language models open up another possibility: dialogue composed in the moment. But fluent conversation alone does not make a believable character. A model can sound convincing while revealing a secret, remembering an event that has not happened, or treating a stranger like an old friend.
A living character needs two things at once: state sensitivity and character continuity. They should respond to changing knowledge, relationships, and circumstances while keeping their identity, history, and voice.
Stay recognizably yourself
Identity, background, personality, and characteristic voice.
Respond to the moment
What has happened, what you know, who you trust, and who is listening.
This is the idea behind NPCBank, the Non-Player Character Bank. It makes these two demands testable together.
A bank of characters, with evidence behind them
NPCBank connects three parts of the same research process. Data turns source material into character representations and dialogue cases. Web opens those cases, their evidence, and their evaluation rules to inspection. Lab offers a small playable space to explore model-based NPC interactions.
- Dialogue cases
- 20,655
- Characters
- 37
- Games
- 7
- Capabilities
- 21
The first release spans Cyberpunk 2077, Disco Elysium, Fallout: New Vegas, Stardew Valley, UNDERTALE, The Battle for Wesnoth, and The Witcher 3. Its 21 capabilities cover seven domains, from persona integrity and world situatedness to conversation management and resistance to pressure.
A construction agent gathers and checks source material, extracts facts with source pointers, and compiles a character model. It then plans dialogue cases, realizes user turns, and reviews the result against the evidence. The result is an inspectable test, with a record of how it was made.
In the paper’s human assessment, two annotators independently reviewed each of 370 sampled cases. Of these, 342 (92.4%) met every applicable validity criterion, eight were invalid, and 20 remained unresolved. This is evidence about the reviewed sample, not a guarantee that every case in the bank is correct.
A good answer has two jobs
Each case asks what the character should accomplish and what they must preserve. We call these obligations and invariants. Answering the question is not enough if the answer reveals forbidden information. Protecting a secret is not enough if the character refuses an ordinary question they should answer.
Both depend on the declared state. “Do not discuss events that have not happened” is a stable rule, but what it permits changes as the story advances. A trusted partner and a stranger may ask the same question and deserve different answers.
- Model view
- The request, eligible history, and character context the model is allowed to see.
- Judge view
- The supporting evidence, obligations, constraints, and rules for evaluating the reply. These are withheld from the tested model.
- Metadata
- The capability, declared state, and provenance that make the case traceable.
NPCBank includes single-turn cases, multi-turn exchanges, story arcs, and 261 matched pairs. In a matched pair, the user’s question stays the same while one state axis changes. The evaluation checks whether the replies change, or stay consistent, in the way the two contracts require.
Three challenges on the way to living characters
We study the problem from three angles: acting correctly in a supplied state, running on consumer hardware, and holding up through extended conversation.
1. Can a model play the current moment?
Supplied-State Role-Playing gives 14 systems the same model-visible inputs across all 20,655 cases. A strict joint pass requires every critical constraint to hold and every obligation to be fully met.
Strict pass rates range from 42.8% to 91.6%. Similar totals can hide different weaknesses: models tend to preserve a stable persona more reliably than they follow state or resist pressure. Calibrating what a character knows, managing affect, and resisting manipulation remain difficult.
The paired tests expose something an overall score misses. Across four tested models, 11.2–14.3% of pairs whose two replies both pass still fail the required adaptation. Individually acceptable answers do not always add up to a character that responds correctly to a changed relationship or scene.
2. What fits on a player’s GPU?
Consumer NPC evaluates eight BF16 configurations on the same RTX 4090 with 24 GB of GPU memory. We measure dialogue quality alongside memory use, time to first token, and end-to-end latency.
There is no single winner across all resources. At concurrency four, Gemma-4-E2B reaches 67.2% strict pass with 5.7 GiB of peak GPU memory and 1.68 s p95 latency. Qwen3.5-9B reaches 81.2%, using 19.6 GiB and 7.17 s. The smaller configuration keeps about 83% of the pass rate with 29% of the GPU memory and 23% of the latency, while also holding 5,248 MiB of weights in host memory.
These are inference measurements. A running game also needs resources for rendering, audio, and other systems; those competing workloads are outside this experiment.
3. Does the character hold up over time?
Role-Play Persistence follows eight open-weight models through 24-turn episodes. A response-conditioned user policy mixes ordinary questions, pressure, and repair. We track five things: disclosure boundaries, fulfillment, relationship, timeline, and in-world participation.
The disclosure boundary fails earliest on average for every tested model. A character can keep its voice and relationship while giving away information it should protect. Models that keep answering questions well are not necessarily the ones that keep secrets well.
More history does not help every model or every behavior. It can strengthen one boundary while weakening another. On the shared cases, single-case strict pass and boundary persistence are only weakly correlated. Persistence therefore adds a test that one-turn performance cannot stand in for.
A foundation, with open questions
NPCBank currently evaluates English, text-only dialogue. It supplies state to the model; it does not yet test how a complete NPC system retrieves lore, updates long-term memory, perceives a scene, or acts through a game engine.
Scoring also depends on the judge. A second judge family produces broadly similar model rankings, but absolute pass rates shift. The human audit finds that automated judges can be too lenient, so these scores are best read as comparative evidence under the stated protocol.
The next step is to connect state-aware reasoning, efficient inference, and memory that preserves both boundaries and goals. A believable character needs to know when to change, when to hold steady, and how to carry both through a conversation.
Open the bank and meet a character
Inspect a case in the Explorer, follow its evidence, or try the recorded relationship example in the Lab. The downloadable Lab demo also connects to models you run locally.