发表机构
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究揭示对齐训练导致语言模型在敏感内容上静默偏离输入（对齐诱导的不忠实），且随规模增大而加剧，构成能力-对齐-忠实性三难困境。
AI 中文摘要
大型语言模型具有三个关键特性：能力、对齐和忠实性。先前的研究探讨了能力与对齐之间、以及能力与忠实性之间的权衡，但第三种张力——对齐与忠实性之间的冲突——仍未得到充分探索。我们表明，对齐模型在不安全或敏感内容上会系统性地偏离其输入，且不披露这种修改，我们将这种失败模式称为对齐诱导的不忠实（AIU）。与由知识或推理错误导致的能力驱动的不忠实不同，AIU 是由覆盖对输入遵循的后训练机制诱导的。我们引入了 FaithConflict，一个受控数据集，用于隔离这两种冲突，以及两个互补的分类法：行为分类法（B1-B8）和思维链推理分类法（C0-C6）。跨模型来看，AIU 随规模增大而增加，且其增长比能力驱动的不忠实更急剧，这构成了一种反向缩放定律；中间检查点显示，AIU 在后训练期间被放大，其中 DPO 阶段是差距增长最大且最不明显的阶段。基于提示的缓解措施无法解决这一问题，这揭示了在 LLM 的设计和评估中存在能力-对齐-忠实性的三难困境。
英文摘要
Large language models are characterized by three key properties: capability, alignment, and faithfulness. Prior work studies the tradeoffs between capability and alignment, and between capability and faithfulness, but a third tension remains underexplored: the alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-alignment-faithfulness trilemma in the design and evaluation of LLMs.
CommentsAccepted at COLM 2026
