arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.11252cs.AI
收藏

超越模型扩展:用于高效深度推理的测试时干预

Beyond Model Scaling: Test-Time Intervention for Efficient Deep Reasoning

South China University of Technology(华南理工大学);Pazhou Laboratory(琶洲实验室);Peng Cheng Laboratory(鹏城实验室)

Qianyue Wang, Jinwu Hu, Yufeng Wang, Huanxiang Lin, Bolin Chen, Zhiquan Wen, Yaofo Chen, Mingkui Tan

AI总结:

Think-with-Me通过测试时干预提升深度推理效率,实现准确性和推理长度的平衡,优于现有方法。

摘要:

大型推理模型(LRMs)在多步推理方面表现优异,但常常面临低效的推理过程,如过度思考和超调,其中过度或偏离方向的推理会增加计算成本并降低性能。现有的高效推理方法采用闭环方式运作,缺乏外部干预机制来引导推理过程。为了解决这个问题,我们提出了Think-with-Me,一种新的测试时交互推理范式,它在推理过程中引入了外部反馈干预。我们的关键见解是,过渡连接词作为自然的干预点,信号自我验证或探索的阶段,并适当使用过渡词来延长推理过程可以提高性能,而过度使用则会影响性能。基于这些见解,Think-with-Me在这些点暂停推理以获取外部反馈,适应性地延长或终止推理以减少冗余并保持准确性。反馈是通过多标准评估(合理性与完整性)生成的,并且来自人类或LLM代理。我们使用组相对策略优化(GRPO)训练目标模型以适应这种交互模式。实验表明,Think-with-Me在有限上下文窗口下实现了准确性和推理长度之间的更优平衡。在AIME24上,Think-with-Me在8K窗口下比QwQ-32B在准确性上高出7.19%,同时将平均推理长度减少了81%。该范式也受益于安全和创造性任务。

英文摘要:

Large Reasoning Models (LRMs) excel at multi-step reasoning but often suffer from inefficient reasoning processes like overthinking and overshoot, where excessive or misdirected reasoning increases computational cost and degrades performance. Existing efficient reasoning methods operate in a closed-loop manner, lacking mechanisms for external intervention to guide the reasoning process. To address this, we propose Think-with-Me, a novel test-time interactive reasoning paradigm that introduces external feedback intervention into the reasoning process. Our key insights are that transitional conjunctions serve as natural points for intervention, signaling phases of self-validation or exploration and using transitional words appropriately to prolong the reasoning enhances performance, while excessive use affects performance. Building on these insights, Think-with-Me pauses reasoning at these points for external feedback, adaptively extending or terminating reasoning to reduce redundancy while preserving accuracy. The feedback is generated via a multi-criteria evaluation (rationality and completeness) and comes from either human or LLM proxies. We train the target model using Group Relative Policy Optimization (GRPO) to adapt to this interactive mode. Experiments show that Think-with-Me achieves a superior balance between accuracy and reasoning length under limited context windows. On AIME24, Think-with-Me outperforms QwQ-32B by 7.19% in accuracy while reducing average reasoning length by 81% under an 8K window. The paradigm also benefits security and creative tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑