arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22556cs.LGcs.AIcs.CL
收藏

通过优势塑造和长度感知梯度调节实现稳定的自适应思维

Stable Adaptive Thinking via Advantage Shaping and Length-Aware Gradient Regulation

Beihang University(北航);Shanghai AI Laboratory(上海人工智能实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Renmin University of China(中国人民大学)

Zihang Xu, Haozhi Xie, Ziqi Miao, Wuxuan Gong, Chen Qian, Lijun Li

AI总结:

本文提出了一种两阶段框架,通过混合微调和自适应强化学习,结合正确性保持优势塑造和长度感知梯度调节,提升大型推理模型在低复杂度查询中的稳定自适应推理能力。

摘要:

大型推理模型(LRMs)通过扩展的推理轨迹实现强大的性能,但它们在处理低复杂度查询时常常表现出过度思考的行为。现有研究试图缓解这一问题,但从根本上受限于准确率-效率权衡的不稳定性以及对异质推理行为的鲁棒性差。为了解决这些挑战,我们提出了一种用于LRMs中稳定自适应思维的两阶段框架。该框架首先应用混合微调,使模型暴露于思考和不思考行为,建立良好的初始化条件。然后进行自适应强化学习,采用正确性保持优势塑造(CPAS)以避免压制正确的长链推理,并采用长度感知梯度调节(LAGR)以在严重推理长度异质性下稳定优化。在Qwen2.5-1.5B和7B上的广泛实验显示,与强大的基线相比,方法表现出一致的改进,准确率提高达+3.7/+3.6个点,同时生成的token数量减少40.6%/43.9%。进一步分析在不同问题难度和分布外任务上的表现,证实了该方法的鲁棒性和泛化能力。

英文摘要:

Large reasoning models (LRMs) achieve strong performance through extended reasoning traces, but they often exhibit overthinking behavior for low-complexity queries. Existing efforts to mitigate this issue are fundamentally limited by unstable accuracy-efficiency trade-offs and poor robustness to heterogeneous reasoning behaviors. To address these challenges, we propose a two-stage framework for stable adaptive thinking in LRMs. The framework first applies Hybrid Fine-Tuning to expose the model to both thinking and no-thinking behaviors, establishing well-conditioned initialization. It then performs adaptive reinforcement learning with Correctness-Preserving Advantage Shaping (CPAS) to avoid suppressing correct long-chain reasoning, and Length-Aware Gradient Regulation (LAGR) to stabilize optimization under severe reasoning-length heterogeneity. Extensive experiments on Qwen2.5-1.5B and 7B show consistent improvements over strong baselines, achieving up to +3.7/+3.6 accuracy points while reducing generated tokens by 40.6%/43.9%. Further analyses across varying problem difficulties and out-of-distribution tasks confirm the robustness and generalization of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑