注意力揭示大语言模型推理：预规划与锚定节奏实现细粒度策略优化
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
AI总结：
本文通过注意力机制揭示大语言模型推理中的预规划与锚定节奏，并据此提出三种细粒度强化学习策略，在多种推理任务上取得一致性能提升。
摘要：
大语言模型的推理模式仍然不透明，强化学习通常对整个生成过程应用统一信用分配，模糊了关键步骤与常规步骤的区别。本文将注意力视为一种特权基质，它使大语言模型的内部逻辑变得可读，不仅是计算的副产品，更是推理本身的机械蓝图。我们首先区分局部和全局聚焦信息处理的注意力头，并揭示局部聚焦头在对角线附近产生锯齿状模式，指示短语块，而全局聚焦头则暴露对后续令牌具有广泛下游影响的令牌。我们用两个指标形式化这些：1）窗口平均注意力距离，衡量裁剪窗口内向后注意力的程度；2）未来注意力影响，量化令牌的全局重要性，即其从后续令牌接收的平均注意力。综合来看，这些信号揭示了一种重复的预规划与锚定机制，其中模型首先进行长距离上下文参考以生成一个引导令牌，该令牌立即跟随或与一个组织后续推理的语义锚定令牌重合。利用这些见解，我们引入了三种新颖的强化学习策略，动态地对关键节点（预规划令牌、锚定令牌及其时间耦合）进行目标信用分配，并在各种推理任务中展示了一致的性能提升。通过将优化与模型的内在推理节奏对齐，我们旨在将不透明的优化转化为可操作的结构感知过程，希望为更透明和有效的大语言模型推理优化提供潜在一步。
英文摘要：
The reasoning pattern of Large language models (LLMs) remains opaque, and reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the distinction between pivotal and routine steps. This work positions attention as a privileged substrate that renders the internal logic of LLMs legible, not merely as a byproduct of computation, but as a mechanistic blueprint of reasoning itself. We first distinguish attention heads between locally and globally focused information processing and reveal that locally focused heads produce a sawtooth pattern near the diagonal indicating phrasal chunks, while globally focused heads expose tokens that exert broad downstream influence over future tokens. We formalize these with two metrics: 1) Windowed Average Attention Distance, which measures the extent of backward attention within a clipped window; 2) Future Attention Influence, which quantifies a token's global importance as the average attention it receives from subsequent tokens. Taken together, these signals reveal a recurring preplan-and-anchor mechanism, where the model first performs a long-range contextual reference to generate an introductory token, which is immediately followed by or coincides with a semantic anchor token that organizes subsequent reasoning. Leveraging these insights, we introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes (preplan tokens, anchor tokens, and their temporal coupling) and show consistent performance gains across various reasoning tasks. By aligning optimization with the model's intrinsic reasoning rhythm, we aim to transform opaque optimization into an actionable structure-aware process, hoping to offer a potential step toward more transparent and effective optimization of LLM reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
