arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.09953cs.CL
收藏

ATTNPO:基于注意力的高效推理过程监督

ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司);Tianjin University(天津大学)

Shuaiyi Nie, Siyu Ding, Wenyuan Zhang, Linhao Yu, Tianmeng Yang, Yao Chen, Weichong Yin, Yu Sun, Hua Wu, Tingwen Liu

AI总结:

本文提出ATTNPO,通过利用模型内在注意力信号进行步骤级信用分配,有效减少推理长度并提升性能,适用于9个基准测试。

摘要:

大型通过强化学习和可验证奖励(RLVR)训练的推理模型在复杂推理任务中表现优异,但往往过度思考,生成冗余推理而无性能提升。现有轨迹级长度惩罚通常无法有效缩短推理长度并降低准确性,因为它们统一处理所有推理步骤,缺乏细粒度信号来区分冗余与必要。同时,过程监督方法通常资源消耗大且存在信用分配不准确的问题。为了解决这些问题,我们提出ATTNPO,一种低开销的过程监督RL框架,利用模型内在的注意力信号进行步骤级信用分配。我们首先识别一组特殊的注意力头,这些头自然聚焦于关键步骤并抑制冗余步骤。通过利用这些头的注意力分数,我们然后采用两种子策略来缓解过度思考:通过抑制冗余步骤并减少对关键步骤的惩罚来保持准确性。实验结果表明,ATTNPO在9个基准测试中显著减少了推理长度并大幅提升了性能。

英文摘要:

Large reasoning models trained with reinforcement learning and verifiable rewards (RLVR) achieve strong performance on complex reasoning tasks, yet often overthink, generating redundant reasoning without performance gains. Existing trajectory-level length penalties often fail to effectively shorten reasoning length and degrade accuracy, as they uniformly treat all reasoning steps and lack fine-grained signals to distinguish redundancy from necessity. Meanwhile, process-supervised methods are typically resource-intensive and suffer from inaccurate credit assignment. To address these issues, we propose ATTNPO, a low-overhead process-supervised RL framework that leverages the model's intrinsic attention signals for step-level credit assignment. We first identify a set of special attention heads that naturally focus on essential steps while suppressing redundant ones. By leveraging the attention scores of these heads, We then employ two sub-strategies to mitigate overthinking by discouraging redundant steps while preserving accuracy by reducing penalties on essential steps. Experimental results show that ATTNPO substantially reduces reasoning length while significantly improving performance across 9 benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑