大语言模型的是非偏差反映答案顺序和措辞,而非道德判断的转变
The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment
AI总结:
研究大语言模型在道德困境判断中的是非偏差,通过交叉对称化等心理测量方法区分逻辑判断、词汇等因素,发现前沿模型立场与形式无关,部分模型存在顺序和词汇偏差,最小模型可总结伪像,强调跨框架测量模型重视内容。
摘要:
大语言模型(LLMs)越来越多地做出二元判断,且越来越多文献报道此类判断在措辞的逻辑无关变化下会发生转变,其中在道德困境上存在放大的是非偏差,这在人类中不存在。单个框架无法说明这种转变是什么,因为在是/否问题中,“否”这个词既是逻辑判断、词汇单元,又是最后打印的选项。我们引入了一套心理测量方法来区分这些:交叉对称化——在一系列问题形式的语料库中,将每个逻辑无关因素以平衡对的形式翻转。对逻辑等价形式的分级评分恢复了一个连贯的内部道德尺度:前沿模型的立场θ几乎与形式无关(在±1轴上跨形式不一致性为0.12 - 0.21);小型开放权重模型以特定于模型的方式失败。通过是/否覆盖强制得出判断会叠加一个可分解的伪像:对最后打印选项的顺序偏差——与经典的人类首位效应相反——加上对“否”这个词的词汇吸引;这种伪像仅在Claude模型中显著(故事平均为 - 0.32至 - 0.86),GPT - 5.5和Gemini约为0,并且在扩展推理下会缩小。单词和判断共享一个单元;将单词换成任意标签将它们分开,并且对于每个前沿模型,与判断相关的逻辑偏差证明约为0,而特定于模型的标签和顺序关联仍然存在:模型并非倾向于拒绝——这种吸引遵循打印表面,而非其承载的判断。一个最小模型\(P = \sigma((\theta \pm m)/s)\),通过框架敏感性m和道德决定性s总结任何此类伪像,这与采样温度有明显区别。该方法适用于任何困境集和二元格式:衡量模型重视什么需要跨越问题框架,而不是只问一次。
英文摘要:
Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option. We introduce a psychometric battery that separates these: crossed symmetrization - every logically irrelevant factor flipped in balanced pairs - across a corpus of question forms. A graded rating across logically equivalent forms recovers a coherent internal moral scale: frontier models' stance $θ$ is nearly format-invariant (cross-form incoherence 0.12-0.21 on a $\pm 1$ axis); small open-weight models fail in model-specific ways. Forcing the verdict through yes/no overlays a decomposable artifact: an order bias toward the last-printed option - opposite to classic human primacy - plus a lexical pull toward the word "no"; the artifact is substantial only in the Claude models (story-averaged -0.32 to -0.86), $\approx 0$ for GPT-5.5 and Gemini, and shrinks under extended reasoning. The word and the verdict share one token; swapping the words for arbitrary labels separates them, and the verdict-attached logical bias proves $\approx 0$ for every frontier model, while model-specific label and order attachments remain: the models are not drawn toward rejecting - the pull follows the printed surface, not the verdict it carries. A minimal model, $P = σ((θ\pm m)/s)$, summarizes any such artifact by a framing susceptibility m and a moral decisiveness s, measurably distinct from sampling temperature. The battery applies unchanged to any dilemma set and binary format: measuring what a model values requires crossing the frames of the question, not asking once.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。