arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.07572cs.LGcs.AIcs.CL
收藏

通过元强化微调优化测试时计算

Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning

Carnegie Mellon University(卡内基梅隆大学);Hugging Face

Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, Aviral Kumar

AI总结:

本文将测试时计算优化形式化为元强化学习问题,提出以累积遗憾衡量计算效能,并开发元强化微调方法MRT,在数学推理中实现2-3倍性能提升和约1.5倍token效率提升。

摘要:

训练模型有效利用测试时计算对于提升大语言模型(LLM)的推理性能至关重要。当前方法大多通过对搜索轨迹进行微调,或使用0/1结果奖励进行强化学习(RL)来实现,但这些方法能否高效地利用测试时计算?随着预算的增加,这些方法能否持续扩展?本文尝试回答这些问题。我们将优化测试时计算的问题形式化为一个元强化学习(RL)问题,这为如何花费测试时计算提供了一个具有原则性的视角。这一视角使我们能够将LLM的长输出流视为在测试时运行的多个回合(episodes),并引导我们使用输出token上的累积遗憾(cumulative regret)这一概念来衡量测试时计算的效能。类似于RL算法如何在训练过程中最好地权衡探索与利用,最小化累积遗憾也将在token流中提供探索与利用之间的最佳平衡。我们表明,尽管最先进的模型并未最小化遗憾,但可以通过在结果0/1奖励RL的基础上最大化一个密集奖励加成来实现这一目标。该加成是输出流中每个后续块所取得的“进展”,通过最终成功可能性的变化来量化。利用这些见解,我们开发了元强化微调(Meta Reinforcement Fine-Tuning,简称MRT),这是一类用于优化测试时计算的新型微调方法。与结果奖励RL相比,MRT在数学推理上带来了2-3倍的相对性能提升,并在token效率上获得了约1.5倍的提升。

英文摘要:

Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs. Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget improves? In this paper, we try to answer these questions. We formalize the problem of optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a principled perspective on spending test-time compute. This perspective enables us to view the long output stream from the LLM as consisting of several episodes run at test time and leads us to use a notion of cumulative regret over output tokens as a way to measure the efficacy of test-time compute. Akin to how RL algorithms can best tradeoff exploration and exploitation over training, minimizing cumulative regret would also provide the best balance between exploration and exploitation in the token stream. While we show that state-of-the-art models do not minimize regret, one can do so by maximizing a dense reward bonus in conjunction with the outcome 0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in the output stream, quantified by the change in the likelihood of eventual success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑