Task-driven Roleplay Agentic Checklist Evaluation
Roleplay evaluation should reveal what was tested, what failed, and which dialogue evidence supports the judgment. TRACE BENCH turns each role profile into a fixed checklist, then adaptively verifies it through natural multi-turn interaction.
TRACE BENCH actively reaches fixed role requirements and preserves the evidence needed to audit every judgment.
Coverage = (completed + failed) / total. It measures whether an item was tested, not whether the model passed it.
Agreement with human majority over 600 checklist items.
Fleiss' kappa = 0.7255Seeded case-runs contaminated by a fixed first message.
Removed in the final protocolMean CC change under refined verification flows.
Same requirements and denominatorScored requirements are built once from the role profile. The User Agent changes only how those requirements are elicited and verified.
Built once and fixed across models
Rowan, the North Gate sent me. I carry the watch token.
Then the watch trusts you. The archive seals changed after dusk.
C2 -> completed
Evidence: watch token recognized
Planning and tool results never enter public dialogue
Every terminal judgment keeps supporting dialogue evidence
Effective verification actions are distilled from failed traces. Role requirements and the scoring denominator stay fixed.
Verification evolves; requirements do not
Every item must reach a terminal state with evidence before the conversation-finish tool can release the evaluation.
Completed non-STM role requirements over all non-STM prebuilt items.
Cases that complete the cross-turn user-injected fact probe.
Character-bigram sentence similarity mapped to a repetition penalty.
Per-reply LLM judgment of fluency, usage, and internal logic.
Deterministic language-aware bounds for response length.
Results use the same 200 cases and 5,498 fixed checklist items. Overall follows the five-dimension weighted formula; C-to-F is diagnostic only.
The paper reports 100% Covered Rate for the main benchmark and therefore omits it from the leaderboard.
Open full tables and analysisAll six tested models preserve their ordering across three independent runs.
Three qualified User Agents produce exactly the same six-model ordering.
Later counterevidence can overturn success; failure evidence cannot be erased.
Every case packages a target role profile, user profile, interaction scene, and fixed checklist for repeated evaluation across models.
Original public character profiles supplemented with scenes and user-side context.
Chinese-English roles with richer relationships, tasks, and behavioral constraints.
* Equal contribution to technical implementation.