arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.14901cs.LGcs.AIcs.CL
收藏

基于采样的推理:你的基础模型比你想象的更聪明

Reasoning with Sampling: Your Base Model is Smarter Than You Think

Harvard University(哈佛大学)

Aayush Karan, Yilun Du

AI总结:

本文提出一种基于MCMC的迭代采样算法,在推理时无需训练即可从基础模型引出推理能力,在多项任务上接近甚至超越强化学习后训练效果,并避免多样性崩溃。

摘要:

前沿推理模型在众多学科中展现出惊人的能力,这得益于通过强化学习(RL)对大型语言模型(LLMs)进行后训练。然而,尽管这一范式取得了广泛成功,大量文献致力于厘清在RL过程中涌现但基础模型中并不存在的新行为。在我们的工作中,我们从不同角度探讨这一问题,转而询问:是否可以通过纯采样,在推理时从基础模型中引出相当的推理能力,而无需任何额外训练。受用于从锐化分布采样的马尔可夫链蒙特卡洛(MCMC)技术启发,我们提出了一种简单的迭代采样算法,利用基础模型自身的似然。在不同基础模型上,我们表明我们的算法在多种单次任务(包括MATH500、HumanEval和GPQA)中带来了显著的推理提升,几乎匹配甚至超越RL的效果。此外,我们的采样器避免了RL后训练所特有的多样样本多样性崩溃问题。关键在于,我们的方法不需要训练、精选数据集或验证器,表明其可广泛应用于易于验证的领域之外。

英文摘要:

Frontier reasoning models have exhibited incredible capabilities across a wide array of disciplines, driven by posttraining large language models (LLMs) with reinforcement learning (RL). However, despite the widespread success of this paradigm, much of the literature has been devoted to disentangling truly novel behaviors that emerge during RL but are not present in the base models. In our work, we approach this question from a different angle, instead asking whether comparable reasoning capabilites can be elicited from base models at inference time by pure sampling, without any additional training. Inspired by Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened distributions, we propose a simple iterative sampling algorithm leveraging the base models' own likelihoods. Over different base models, we show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL on a wide variety of single-shot tasks, including MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in diversity over multiple samples that is characteristic of RL-posttraining. Crucially, our method does not require training, curated datasets, or a verifier, suggesting broad applicability beyond easily verifiable domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑