arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.06703cs.CL
收藏

1B大语言模型能否超越405B大语言模型?重新思考计算最优的测试时缩放

Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling

Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);BUPT(北京邮电大学)

Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, Bowen Zhou

AI总结:

本文针对测试时缩放(TTS)研究的缺失,通过实验分析策略模型、PRMs和问题难度对TTS的影响,提出计算最优TTS策略,实现小模型(如1B LLM)在推理任务上超越大模型,为提升LLMs推理能力提供新方向。

摘要:

测试时缩放(TTS)是一种通过在推理阶段使用额外计算来提升大语言模型(LLMs)性能的重要方法。然而,现有研究未系统分析策略模型、过程奖励模型(PRMs)和问题难度如何影响TTS。这种分析缺失限制了对TTS方法的理解与实际应用。本文聚焦两个核心问题:(1)在不同策略模型、PRMs和问题难度水平下,缩放测试时计算的最优方法是什么?(2)扩展计算能在多大程度上提升LLMs在复杂任务上的性能,较小的语言模型能否通过该方法超越较大模型?通过在MATH-500和具有挑战性的AIME24任务上的全面实验,我们得到以下观察结果:(1)计算最优的TTS策略高度依赖于策略模型、PRM和问题难度的选择。(2)使用我们的计算最优TTS策略,极小的策略模型能超越较大模型。例如,1B LLM在MATH-500上能超过405B LLM。此外,在MATH-500和AIME24上,0.5B LLM优于GPT-4o,3B LLM超越405B LLM,7B LLM击败o1和DeepSeek-R1,同时具有更高的推理效率。这些发现表明,使TTS策略适应每个任务和模型的特定特征具有重要意义,并指出TTS是提升LLMs推理能力的一种有前景的方法。

英文摘要:

Test-Time Scaling (TTS) is an important method for improving the performance of Large Language Models (LLMs) by using additional computation during the inference phase. However, current studies do not systematically analyze how policy models, Process Reward Models (PRMs), and problem difficulty influence TTS. This lack of analysis limits the understanding and practical use of TTS methods. In this paper, we focus on two core questions: (1) What is the optimal approach to scale test-time computation across different policy models, PRMs, and problem difficulty levels? (2) To what extent can extended computation improve the performance of LLMs on complex tasks, and can smaller language models outperform larger ones through this approach? Through comprehensive experiments on MATH-500 and challenging AIME24 tasks, we have the following observations: (1) The compute-optimal TTS strategy is highly dependent on the choice of policy model, PRM, and problem difficulty. (2) With our compute-optimal TTS strategy, extremely small policy models can outperform larger models. For example, a 1B LLM can exceed a 405B LLM on MATH-500. Moreover, on both MATH-500 and AIME24, a 0.5B LLM outperforms GPT-4o, a 3B LLM surpasses a 405B LLM, and a 7B LLM beats o1 and DeepSeek-R1, while with higher inference efficiency. These findings show the significance of adapting TTS strategies to the specific characteristics of each task and model and indicate that TTS is a promising approach for enhancing the reasoning abilities of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑