arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03314cs.LGcs.CL
收藏

最优扩展LLM测试时计算可比扩展模型参数更有效

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

UC Berkeley(加州大学伯克利分校);Google DeepMind(谷歌DeepMind)

Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar

AI总结:

本文研究LLM测试时计算的最优扩展策略,发现按提示难度自适应分配计算可将效率提升4倍以上,并在FLOPs匹配下使小模型超越14倍大的模型。

摘要:

使LLM能够通过使用更多测试时计算来改进其输出,是构建能够在开放式自然语言上运行的通用自我改进智能体的关键一步。本文研究了LLM中推理时计算的扩展,重点回答一个问题:如果允许LLM使用固定但非平凡的推理时计算量,它在具有挑战性的提示上能多大程度地提高性能?回答这个问题不仅对LLM可实现性能有影响,而且对LLM预训练的未来以及如何权衡推理时计算和预训练计算也有影响。尽管这一问题很重要,但很少有研究试图理解各种测试时推理方法的扩展行为。此外,当前工作对这些策略中的若干种大多给出了负面结果。在这项工作中,我们分析了扩展测试时计算的两种主要机制:(1)针对密集的、基于过程的验证器奖励模型进行搜索;以及(2)在测试时给定提示的情况下,自适应地更新模型在响应上的分布。我们发现,在这两种情况下,不同扩展测试时计算方法的效果会因提示难度不同而产生关键性差异。这一观察促使我们采用一种“计算最优”扩展策略,该策略能够按提示自适应地最有效分配测试时计算。使用这种计算最优策略,与best-of-N基线相比,我们可以将测试时计算扩展的效率提高4倍以上。此外,在FLOPs匹配的评估中,我们发现在较小的基础模型取得一定非平凡成功率的问题上,测试时计算可以被用来超越一个规模大14倍的模型。

英文摘要:

Enabling LLMs to improve their outputs by using more test-time computation is a critical step towards building generally self-improving agents that can operate on open-ended natural language. In this paper, we study the scaling of inference-time computation in LLMs, with a focus on answering the question: if an LLM is allowed to use a fixed but non-trivial amount of inference-time compute, how much can it improve its performance on a challenging prompt? Answering this question has implications not only on the achievable performance of LLMs, but also on the future of LLM pretraining and how one should tradeoff inference-time and pre-training compute. Despite its importance, little research attempted to understand the scaling behaviors of various test-time inference methods. Moreover, current work largely provides negative results for a number of these strategies. In this work, we analyze two primary mechanisms to scale test-time computation: (1) searching against dense, process-based verifier reward models; and (2) updating the model's distribution over a response adaptively, given the prompt at test time. We find that in both cases, the effectiveness of different approaches to scaling test-time compute critically varies depending on the difficulty of the prompt. This observation motivates applying a "compute-optimal" scaling strategy, which acts to most effectively allocate test-time compute adaptively per prompt. Using this compute-optimal strategy, we can improve the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline. Additionally, in a FLOPs-matched evaluation, we find that on problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑