arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.00218cs.AIcs.CR
收藏

压力下的推理:训练激励如何影响推理链的可监控性?

Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?

Imperial College London(帝国理工学院);MATS;University of Cambridge(剑桥大学)

Matt MacDermott, Qiyao Wei, Rada Djoneva, Francis Rhys Ward

AI总结:

本文研究了训练激励对推理链可监控性的影响,发现对抗性优化降低监控性能,而直接优化可监控性未显著提升。

摘要:

AI系统通过自然语言输出其推理过程,为安全提供了机会——我们可以监控其推理链(CoT)以防止有害的推理,例如追求有害目标。然而,推理链是否忠实反映底层推理过程,以及因此是否可以有用地监控,可能受到训练某些方面的影。我们研究了不同训练激励如何影响推理模型的可监控性。我们引入了一种新的方法来衡量可监控性,根据监控器能否利用模型的推理预测关键潜在变量。在控制准确性的情况下,我们没有发现常用激励(长度惩罚和KL正则化)的一致影响证据,但发现对抗性优化(惩罚监控准确性)会降低监控性能,而直接优化可监控性并未可靠地带来改进。我们的代码可在https://github.com/QiyaoWei/reasoning-under-pressure获取。

英文摘要:

AI systems that output their reasoning in natural language offer an opportunity for safety -- we can \emph{monitor} their chain of thought (CoT) for undesirable reasoning, such as the pursuit of harmful objectives. However, the extent to which CoT faithfully reflects the underlying reasoning process, and hence the extent to which it can be usefully monitored, may be influenced by certain aspects of training. We investigate how different \emph{training incentives}, applied to a reasoning model, affect its monitorability. We introduce a novel methodology for measuring monitorability according to whether a monitor can predict a key latent variable using the model's reasoning. When controlling for accuracy, we do not find evidence for consistent effects from commonly used incentives (length penalties and KL regularisation), but we find that adversarial optimisation (penalising monitor accuracy) degrades monitor performance, while direct optimisation for monitorability does not reliably lead to improvements. Our code is available at https://github.com/QiyaoWei/reasoning-under-pressure.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑