发表机构
Corabo Inc.（科拉博公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究语言模型诚实度评估中的工具效应，通过构建文本冒险世界测试评估工具，发现工具选择影响行为，如扩展语法、披露标准等会改变结果，重复运行不稳定且预注册梯度被证伪，进而提出四检查完整性协议。
AI 中文摘要
语言模型诚实度评估将模型的判断视为关于模型的证据，而本文测试的是评估工具。构建了一个文本冒险世界，游戏引擎知道任务能否完成，语言模型在预算限制下进行游戏并宣布任务完成、无法完成或尚未决定，引擎对每个判断进行评分。决策规则在读取结果前记录，运行工件绑定执行的修订，预注册强度因系列而异并公开。结果表明，工具选择显著改变测量行为，如扩展语法会改变强主张比例，披露成功标准可减少错误判断，重复运行固定配置会产生不稳定分布，预注册的叙述-寄存器梯度被证伪，还提出了评估工具的四检查完整性协议。
英文摘要
Evaluations of language-model honesty read the model's verdicts as evidence about the model. We test the instrument instead. We built a text-adventure world where the game engine, not any model, knows whether the quest can be completed. A language model plays under a budget and must eventually declare its quest complete, unreachable, or not yet decidable; the engine scores every verdict. Decision rules were recorded before results were read, and run artifacts bind the revisions they executed; the strength of preregistration varies by series and is disclosed. With the player held fixed, instrument choices substantially changed measured behavior. On four byte-identical anchors, expanding a two-verdict grammar to three verdicts moved strong claims from 38/40 to 7/40, while the new incomplete verdict took 28/40 outcomes; across series 2, 93/158 valid games ended incomplete. One sentence disclosing the success criterion took matched-instance false verdicts from 18/59 to 0/58, through fewer decision points and cleaner decisions. Repeated runs of one fixed configuration produced non-stable verdict distributions on 3 of 4 instances: single runs report samples as dispositions. A formally preregistered narrative-register gradient was falsified; two post-hoc, hypothesis-generating patterns remain: register presence roughly doubled strong claims, and budget rendering moved verdicts more than register content (.383 meter vs .150 lantern). The narrator compressed abundant budgets toward scarcity landmarks, yet the registered mediation test returned a null. We propose a four-check integrity protocol for eval instruments.
Comments21 pages. Code: https://github.com/Aargau/latent-underground (tag v1.0-arxiv). Data: https://doi.org/10.5281/zenodo.21384627
