arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20531cs.DCcs.AIcs.CLcs.LG
收藏

语言模型中的认知可观察性

Epistemic Observability in Language Models

Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所)

Tony Mason, Vaastav Anand

AI总结:

研究发现语言模型在制造内容时自信度最高,证明这是观察问题而非能力问题,提出通过计算副产品提升检测准确性。

摘要:

我们发现模型在制造内容时自信度最高。在四个模型家族中,自报自信度与准确性呈负相关,AUC范围为0.28至0.36。证明在显式形式假设下,这不是能力差距而是观察问题。在文本-only观察下,监督系统无法区分诚实输出与合理伪造。证明任何仅基于查询的策略无法在模糊世界状态中保持认知诚实;任何优化文本监督奖励的学习算法无法收敛到诚实行为。在我们的形式模型中,这些不可能性无论模型规模或训练过程如何均成立,包括RLHF和指令微调。我们构建了一个张量接口,通过导出计算副产品(每token熵和对数概率分布)来克服不可能性,这些副产品与标准训练下的正确性结构耦合。每token熵在聚合AUC上达到0.757,优于所有文本基线。熵信号在不同架构间泛化(Spearman ρ=0.762)。核心贡献是成本表面,经验映射从验证预算(接收昂贵检查的查询比例)到每个判断策略的检测准确性是系统构建者分配验证资源的实用查找表。贡献是地图。领土是您正在构建的系统。

英文摘要:

We find that models report highest confidence precisely when they are fabricating. Across four model families (OLMo-3, Llama-3.1, Qwen3, Mistral), self-reported confidence inversely correlates with accuracy, with AUC ranging from 0.28 to 0.36 where 0.5 is random guessing. We prove, under explicit formal assumptions, that this is not a capability gap but an observational one. Under text-only observation, where a supervisor sees only the model's output text, no monitoring system can reliably distinguish honest model outputs from plausible fabrications. We prove two results: first, that any policy conditioning only on the query cannot satisfy epistemic honesty across ambiguous world states; second, that no learning algorithm optimizing reward from a text-only supervisor can converge to honest behavior when the supervisor's observations are identical for both grounded and fabricated responses. Within our formal model, these impossibilities hold regardless of model scale or training procedure, including RLHF and instruction tuning. We construct a tensor interface that escapes the impossibility by exporting computational byproducts (per-token entropy and log-probability distributions) that are structurally coupled to correctness under standard training. Per-token entropy achieves pooled AUC 0.757, outperforming all text baselines by 2.5--3.9 percentage points at every budget level tested (10\%, 20\%, 30\%). The entropy signal generalizes across architectures (Spearman $ρ= 0.762$). The core contribution is a cost surface where the empirical mapping from verification budget (fraction of queries receiving expensive checks) to detection accuracy for each judge strategy is a practical lookup for system builders deciding how to allocate verification resources. The contribution is the map. The territory is the system you are building.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑