arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.07191cs.AI
收藏

用于 LLM 推理的推理时计算技巧集锦

Bag of Tricks for Inference-time Computation of LLM Reasoning

Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Didichuxing Co. Ltd(滴滴出行有限公司)

Fan Liu, Wenshuo Chao, Naiqiang Tan, Hao Liu

AI总结:

本文系统基准测试 LLM 推理时计算方法,从候选生成与奖励机制两方面消融优化,发现调温等技巧可显著提升性能,并建立覆盖六方法八任务的标准化基准。

摘要:

随着大型语言模型(LLM)的发展,解决复杂推理任务受到越来越多关注。推理时计算方法(例如 Best-of-N、束搜索等)尤其有价值,因为它们可以在不修改模型参数或不需要额外训练的情况下提升推理性能。然而,这些技术存在实现上的挑战,而且由于计算复杂度以及在不同任务上效果不一,大多数现有方法仍停留在概念验证阶段,实际应用有限。本文在不同复杂度的推理任务上研究并基准测试多种推理时计算策略。由于当前多数方法依赖 proposer-verifier(提议者—验证者)流程:先生成候选解(例如推理解),再基于奖励信号(例如 RLHF 奖励、过程奖励)选择最佳解,因此我们的研究重点是同时优化候选解生成(例如指令提示,以及 temperature 和 top-p 等超参数)与奖励机制(例如自评估、奖励类型)。通过在多种规模的各类模型(例如 Llama、Qwen 和 Mistral 系列)上开展大量实验(超过 20,000 个 A100-80G GPU 小时,包含 1,000 多项实验),我们的消融研究表明,以往被忽视的策略能够显著提升性能(例如调节 temperature 可使推理任务性能最高提升 5%)。此外,我们通过在八个推理任务上系统评估六种代表性方法,建立了一个用于推理时计算的标准化基准。这些发现为未来研究提供了更坚实的基础。代码可在 https://github.com/usail-hkust/benchmark_inference_time_computation_LLM 获取。

英文摘要:

With the advancement of large language models (LLMs), solving complex reasoning tasks has gained increasing attention. Inference-time computation methods (e.g., Best-of-N, beam search, et al.) are particularly valuable as they can enhance reasoning performance without modifying model parameters or requiring additional training. However, these techniques come with implementation challenges, and most existing methods remain at the proof-of-concept stage with limited practical adoption due to their computational complexity and varying effectiveness across different tasks. In this paper, we investigate and benchmark diverse inference-time computation strategies across reasoning tasks of varying complexity. Since most current methods rely on a proposer-verifier pipeline that first generates candidate solutions (e.g., reasoning solutions) and then selects the best one based on reward signals (e.g., RLHF rewards, process rewards), our research focuses on optimizing both candidate solution generation (e.g., instructing prompts, hyperparameters such as temperature and top-p) and reward mechanisms (e.g., self-evaluation, reward types). Through extensive experiments (more than 20,000 A100-80G GPU hours with over 1,000 experiments) across a variety of models (e.g., Llama, Qwen, and Mistral families) of various sizes, our ablation studies reveal that previously overlooked strategies can significantly enhance performance (e.g., tuning temperature can improve reasoning task performance by up to 5%). Furthermore, we establish a standardized benchmark for inference-time computation by systematically evaluating six representative methods across eight reasoning tasks. These findings provide a stronger foundation for future research. The code is available at https://github.com/usail-hkust/benchmark_inference_time_computation_LLM

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑