arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.07660cs.CL
收藏

并非所有令牌都以相同方式学习:注意力熵揭示了强化学习推理中的异质信号

Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning

Baidu(百度);School of Computer Science, Peking University(北京大学计算机科学系);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院)

Gengyang Li, Zheng-Fan Wu, Siqi Bao, Yunfang Wu

AI总结:

研究通过注意力熵揭示强化学习推理中令牌层面的异质性,发现高熵令牌(探索者)与低熵令牌(锚点)在学习信号上有显著差异,动态熵感知的软重加权干预提升了模型性能。

摘要:

基于强化学习的后训练方法已成为提升大语言模型推理能力的关键途径,但其令牌层面的学习信号仍缺乏深入理解。本文通过注意力熵研究其异质性,发现令牌层面的强化学习目标稀疏可估计:随机20%令牌子集能保持大部分全令牌留出性能,表明令牌层面更新存在大量冗余。然而,熵结构化的子集行为差异显著。低注意力熵令牌(锚点)依赖集中支持,产生与全令牌更新对齐的稳定梯度,提供可靠优化基础,但倾向于在更难的基准上停滞。高注意力熵令牌(探索者)聚合更多分散上下文,诱导更大但更波动的梯度。仅训练探索者在平均上不稳定,尽管偶尔成功运行表明这些令牌可能在优化稳定时包含有用的硬推理信号。本文通过证据收集分析、熵动态、梯度几何诊断和控制实验支持锚点-探索者光谱,证明位置、预测熵和损失归一化无法解释观察到的不对称性。最终,动态熵感知的软重加权干预将Qwen3-8B-Base在最强设置下的留出平均提升至37.40,这些发现表明注意力熵揭示了令牌层面强化学习信号中的优化相关结构,且统一令牌平均可能掩盖推理后训练中的有意义异质性。

英文摘要:

Reinforcement-learning-based post-training has become a key approach for improving the reasoning ability of large language models, but its token-level learning signals remain poorly understood. This work studies their heterogeneity through attention entropy, which measures how concentrated or diffuse the contextual support is for each response token. We first show that token-level RL objectives are sparsely estimable: uniformly random 20 percent token subsets preserve much of the full-token held-out performance, suggesting substantial redundancy in token-level updates. However, entropy-structured subsets behave very differently. Low-attention-entropy tokens, which we call anchors, rely on concentrated support, produce stable gradients aligned with full-token updates, and provide a reliable optimization backbone, but tend to plateau on harder benchmarks. High-attention-entropy tokens, which we call explorers, aggregate more diffuse context and induce larger but more volatile gradients. Explorer-only training is unstable on average, though rare successful runs suggest that these tokens may contain useful hard-reasoning signals when optimization remains stable. We support this anchor-explorer spectrum with evidence-gathering analyses, entropy dynamics, gradient-geometry diagnostics, and controls showing that position, predictive entropy, and loss normalization do not explain the observed asymmetry. Finally, a dynamic entropy-aware soft-reweighting intervention improves Qwen3-8B-Base from 34.39 to 37.40 held-out average in the strongest setting. These findings suggest that attention entropy reveals optimization-relevant structure in token-level RL signals, and that uniform token averaging can obscure meaningful heterogeneity in reasoning post-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑