arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.01581cs.LGcs.AIcs.CL
收藏

Think Right:通过自适应注意力压缩学习缓解思考不足与过度问题

Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

UNC Chapel Hill(北卡罗来纳大学教堂山分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Microsoft Research(微软研究院)

Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad, Elias Stengel-Eskin, Akshay Nambi, Mohit Bansal

AI总结:

针对大模型推理中存在的思考不足、过度及自适应不足问题,提出基于自适应注意力压缩的在线后训练RL方法TRAAC,可按任务难度分配推理预算,在多任务上实现准确率提升与推理长度下降,且泛化性强。

摘要:

近期的思考模型通过扩展测试时计算量来解决复杂推理任务,但这种计算扩展必须与任务难度相匹配。一方面,短推理(思考不足)会在需要更多推理步骤的难题上导致错误;但另一方面,过长的推理(思考过度)可能存在token效率低下的问题,甚至在得出正确的中间解后仍会生成不必要的步骤。我们将这种现象称为自适应不足,即模型无法根据不同难度的问题适当调整其响应长度。为解决自适应不足问题并在思考不足与过度之间取得平衡,我们提出TRAAC(Think Right with Adaptive, Attentive Compression,基于自适应注意力压缩的正确思考方法),这是一种在线训练后强化学习方法,利用模型对长推理轨迹的自注意力来识别重要步骤并剪枝冗余步骤。TRAAC还会估计难度并将其纳入训练奖励,从而学会根据样本难度分配相匹配的推理预算。与基础模型和其他强化学习基线相比,我们的方法提升了准确率,减少了推理步骤,并实现了自适应思考。在多种任务(AIME、AMC、GPQA-D、BBEH)上,TRAAC(基于Qwen3-4B)与基础模型相比,平均绝对准确率提升8.4%,推理长度相对减少36.8%;与最优强化学习基线相比,准确率提升7.9%,长度下降29.4%。TRAAC还展现出强泛化性:尽管我们的模型在数学数据集上训练,但在分布外的非数学数据集(如GPQA-D、BBEH和OptimalThinkingBench)上也实现了准确率和效率的提升。我们的分析进一步验证,TRAAC能根据难度对思考预算进行细粒度调整,且任务难度校准与基于注意力的压缩相结合可在各类任务上带来增益。

英文摘要:

Recent thinking models are capable of solving complex reasoning tasks by scaling test-time compute, but this scaling should be allocated in line with task difficulty. On one hand, short reasoning (underthinking) leads to errors on harder problems that require extended reasoning steps; but, excessively long reasoning (overthinking) can be token-inefficient by generating unnecessary steps even after reaching a correct intermediate solution. We refer to this as under-adaptivity, where the model fails to modulate its response length appropriately given problems of varying difficulty. To address under-adaptivity and strike a balance between under- and overthinking, we propose TRAAC (Think Right with Adaptive, Attentive Compression), an online post-training RL method that leverages the model's self-attention to identify key steps and prune redundant ones. TRAAC also estimates difficulty and incorporates it into training rewards, thereby learning to allocate a reasoning budget commensurate with example difficulty. Across a variety of tasks (AIME, AMC, GPQA-D, BBEH), TRAAC (Qwen3-4B) achieves an average absolute accuracy gain of 8.4% with a relative reduction in reasoning length of 36.8% compared to the base model, and a 7.9% accuracy gain paired with a 29.4% length drop compared to the best RL baseline. TRAAC generalizes well, with accuracy and efficiency gains on out-of-distribution non-math datasets like GPQA-D, BBEH, and OptimalThinkingBench. Our analysis shows that TRAAC learns to adjust its thinking budget based on difficulty and that a combination of task-difficulty calibration and attention-based compression yields gains across diverse tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑