认知失调：为什么语言模型输出与真实性的内部表征不一致？
Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?
AI总结：
本文分析语言模型输出查询与内部真实性探测之间的分歧，将其归为虚构、欺骗和异质性三类，并指出探测优势常来自校准与互补表现，集成二者可提升准确率。
摘要：
神经语言模型（LM）可通过两种方式评估事实陈述的真实性：查询陈述概率，或探测其内部真实性表征。以往研究发现这两种程序有时会产生分歧，且探测往往比LM输出更准确，这使部分研究者认为LM会“说谎”或编码了非合作的交流意图。这是对当今LM的准确描述，还是查询—探测分歧可能由其他原因产生？我们识别出三类不同的分歧，分别称为虚构、欺骗和异质性。在许多情况下，探测的优越性仅源于对不确定答案更好的校准，而非正确且高置信答案的比例更高。有时查询和探测在不同输入子集上表现更佳，将二者集成还可进一步提升准确率。代码见github.com/lingo-mit/lm-truthfulness。
英文摘要：
Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal representations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents. Is this an accurate description of today's LMs, or can query-probe disagreement arise in other ways? We identify three different classes of disagreement, which we term confabulation, deception, and heterogeneity. In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers. In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two. Code is available at github.com/lingo-mit/lm-truthfulness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
