arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.05706cs.AI
收藏

推理模型难以控制其推理链

Reasoning Models Struggle to Control their Chains of Thought

NYU(纽约大学);MATS(马歇尔应用科学与技术学院);UCL(伦敦大学学院);UPenn(宾夕法尼亚大学);OpenAI

Chen Yueh-Han, Robert McCarthy, Bruce W. Lee, He He, Ian Kivlichan, Bowen Baker, Micah Carroll, Tomek Korbak

AI总结:

研究发现推理模型的CoT可控性低于输出可控性,且受模型规模、训练和问题难度影响,建议未来跟踪CoT可控性以维持监控有效性。

摘要:

推理链(CoT)监控是一种检测现代推理模型误行为和理解其动机的有希望的工具。然而,如果模型能够控制其在推理链中所表达的内容,这可能会削弱CoT的可监控性。为衡量这种不利能力——CoT可控性,我们引入了CoT-控制评估套件,其中包括要求模型在遵守CoT指令的情况下解决问题的任务,例如在不使用'染色体'一词的情况下推理遗传学问题。我们发现,推理模型的CoT可控性显著低于输出可控性;例如,Claude Sonnet 4.5只能在2.7%的时间内控制其CoT,但控制其最终输出时为61.9%。我们还发现,CoT可控性随着模型规模的增大而提高,随着强化学习训练、测试时计算量和问题难度的增加而降低。CoT可控性失败甚至延伸到模型被激励(而非直接要求)以逃避CoT监控的情况,尽管模型在被告知被监控时表现出略高的可控性。同样,通过对抗性优化提示来激发可控性并不会显著提高可控性。我们的结果使我们谨慎乐观,认为当前CoT可控性不太可能成为CoT可监控性的失败模式。然而,低可控性的机制尚不明确。鉴于其对维持CoT可监控性的重要性,我们建议前沿实验室在未来模型中跟踪CoT可控性。

英文摘要:

Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability -- CoT controllability -- we introduce the CoT-Control evaluation suite, which includes tasks that require models to solve problems while adhering to CoT instructions, e.g., reasoning about a genetics question without using the word 'chromosome'. We show that reasoning models possess significantly lower CoT controllability than output controllability; for instance, Claude Sonnet 4.5 can control its CoT only 2.7% of the time but 61.9% when controlling its final output. We also find that CoT controllability is higher for larger models and decreases with more RL training, test-time compute, and increased problem difficulty. CoT controllability failures extend even to situations in which models are given incentives (as opposed to direct requests) to evade CoT monitors, although models exhibit slightly higher controllability when they are told they are being monitored. Similarly, eliciting controllability by adversarially optimizing prompts does not meaningfully increase controllability. Our results leave us cautiously optimistic that CoT controllability is currently unlikely to be a failure mode of CoT monitorability. However, the mechanism behind low controllability is not well understood. Given its importance for maintaining CoT monitorability, we recommend that frontier labs track CoT controllability in future models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑