arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03978cs.AIcs.LG
收藏

监控性作为免费礼物:RLVR如何自发地对齐推理

Monitorability as a Free Gift: How RLVR Spontaneously Aligns Reasoning

Harvard University(哈佛大学)

Zidi Xiong, Shan Chen, Himabindu Lakkaraju

AI总结:

RLVR通过增强响应分布和提示关注度自发提升推理透明性,但这种提升依赖于数据多样性与训练难度。

摘要:

随着大型推理模型(LRMs)越来越多地被部署,对其思维链(CoT)轨迹进行安全审计变得至关重要。最近的研究报告指出,在强化学习与可验证奖励(RLVR)的早期阶段,监控性——即思维链是否忠实且信息丰富地反映内部计算——可以作为一种“免费礼物”出现。我们通过系统评估不同模型家族和训练领域来使这一观察更加具体。我们的结果表明,这种效果并非普遍:监控性提升是强烈数据依赖的。特别是,我们展示了在RLVR训练中数据多样性和遵循指示的数据的关键作用。我们进一步表明,监控性与能力是相互独立的——推理性能的提升并不意味着透明度的增加。通过机制分析,我们将监控性提升主要归因于响应分布变窄(熵减少)和对提示的更大关注,而不是更强的因果依赖于推理轨迹。我们还揭示了监控性动态如何随着受控训练和评估难度变化。这些发现提供了RLVR下监控性如何出现的全面视图,澄清了何时收益可能发生,何时不会发生。

英文摘要:

As Large Reasoning Models (LRMs) are increasingly deployed, auditing their chain-of-thought (CoT) traces for safety becomes critical. Recent work has reported that monitorability--the degree to which CoT faithfully and informatively reflects internal computation--can appear as a "free gift" during the early stages of Reinforcement Learning with Verifiable Rewards (RLVR). We make this observation concrete through a systematic evaluation across model families and training domains. Our results show that this effect is not universal: monitorability improvements are strongly data-dependent. In particular, we demonstrate the critical role of data diversity and instruction-following data during RLVR training. We further show that monitorability is orthogonal to capability--improvements in reasoning performance do not imply increased transparency. Through mechanistic analysis, we attribute monitorability gains primarily to response distribution sharpening (entropy reduction) and increased attention to the prompt, rather than stronger causal reliance on reasoning traces. We also reveal how monitorability dynamics vary with controlled training and evaluation difficulty. Together, these findings provide a holistic view of how monitorability emerges under RLVR, clarifying when gains are likely to occur and when they are not.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑