arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.01422cs.CLcs.AI
收藏

采样高效的测试时扩展:在早期解码中自我估计Best-of-N采样

Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decoding

School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Tongyi Lab, Alibaba Group(阿里集团通义实验室)

Yiming Wang, Pei Zhang, Siyuan Huang, Baosong Yang, Zhuosheng Zhang, Fei Huang, Rui Wang

AI总结:

提出ST-BoN解码方法,利用模型内部状态的早期采样一致性截断次优路径,无需奖励模型即可避免完全生成所有样本,在保持性能的同时大幅降低GPU内存和延迟开销。

摘要:

测试时扩展通过在推理期间分配额外计算资源来增强大型语言模型性能。Best-of-N (BoN) 采样作为一种常见的基于采样的扩展技术,通过并行拓宽搜索空间从模型分布中寻找更优解。然而,其成本-性能权衡仍有待深入研究。两大主要挑战限制了 BoN 采样的效率:(1) 生成 N 个完整样本会消耗大量 GPU 内存,从而降低有限资源下的推理能力。(2) 奖励模型增加了额外的内存和延迟开销,且训练强大的奖励模型会引入潜在的训练数据成本。尽管部分研究已探索效率提升,但尚无方案能同时解决这两个挑战。为填补该空白,我们提出 Self-Truncation Best-of-N (ST-BoN),这是一种避免完全生成所有 N 个样本并消除对奖励模型需求的解码方法。它利用模型内部状态的早期采样一致性来识别最有希望的路径,并截断次优路径。在成本方面,ST-BoN 将动态 GPU 内存使用量减少超过 80%,将推理延迟降低 50%。在成本-性能权衡方面,ST-BoN 实现了与 Full-BoN 相同的性能,同时节省 70%-80% 的计算成本;在相同成本下,它可以将准确率提高 3-4 个百分点。

英文摘要:

Test-time scaling enhances large language model performance by allocating additional compute resources during inference. Best-of-N (BoN) sampling serves as a common sampling-based scaling technique, broadening the search space in parallel to find better solutions from the model distribution. However, its cost-performance trade-off is still underexplored. Two main challenges limit the efficiency of BoN sampling: (1) Generating N full samples consumes substantial GPU memory, reducing inference capacity under limited resources. (2) Reward models add extra memory and latency overhead, and training strong reward models introduces potential training data costs. Although some studies have explored efficiency improvements, none have addressed both challenges at once. To address this gap, we propose Self-Truncation Best-of-N (ST-BoN), a decoding method that avoids fully generating all N samples and eliminates the need for reward models. It leverages early sampling consistency in the model's internal states to identify the most promising path and truncate suboptimal ones. In terms of cost, ST-BoN reduces dynamic GPU memory usage by over 80% and inference latency by 50%. In terms of cost-performance trade-off, ST-BoN achieves the same performance as Full-BoN while saving computational cost by 70%-80%, and under the same cost, it can improve accuracy by 3-4 points.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑