arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.03149cs.LGcs.DS
收藏

驯服不完美的过程验证器:回溯的采样视角

Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking

Microsoft(微软)

Dhruv Rohatgi, Abhishek Shetty, Donya Saless, Yuchen Li, Ankur Moitra, Andrej Risteski, Dylan J. Foster

AI总结:

针对过程验证器误差会被标准解码放大的问题,本文提出概率回溯的测试时采样算法 VGB,并从随机游走理论与合成、真实语言任务实验两方面证明其更鲁棒且优于基线。

摘要:

将语言模型的生成能力与评估部分生成质量的过程验证器相结合的测试时算法,为激发新的推理能力提供了一种有前景的手段;但这类方法的算法设计空间和计算扩展性质仍不清晰,而且一旦把学习高质量验证器的成本考虑在内,其收益也远非显而易见。我们的出发点是观察到:在生成过程中,由于误差放大,学习到的验证器中看似无害的错误可能导致标准解码技术发生灾难性失败。于是我们追问:能否通过更复杂的解码策略改善这一点?我们提出了一种新的过程引导测试时采样算法 VGB,它采用具有理论依据的回溯,在验证器错误面前实现了可证明更强的鲁棒性。VGB 将自回归生成解释为在部分生成构成的树上进行随机游走,其转移概率由过程验证器和基础模型共同引导;关键在于,回溯以概率方式发生。该过程将理论计算机科学中近似计数与采样文献里具有开创性的 Sinclair-Jerrum 随机游走(Sinclair & Jerrum, 1989)进行了推广,而我们工作的一项概念性贡献,正是强调其与该文献的相似之处。实验上,我们在合成任务和真实语言建模任务上均证明,VGB 在多种指标上优于基线方法。

英文摘要:

Test-time algorithms that combine the generative power of language models with process verifiers that assess the quality of partial generations offer a promising lever for eliciting new reasoning capabilities, but the algorithmic design space and computational scaling properties of such approaches are still opaque, and their benefits are far from apparent when one accounts for the cost of learning a high-quality verifier. Our starting point is the observation that seemingly benign errors in a learned verifier can lead to catastrophic failures for standard decoding techniques due to error amplification during the course of generation. We then ask: can this be improved with more sophisticated decoding strategies? We introduce a new process-guided test-time sampling algorithm, VGB, which uses theoretically grounded backtracking to achieve provably better robustness to verifier errors. VGB interprets autoregressive generation as a random walk on a tree of partial generations, with transition probabilities guided by the process verifier and base model; crucially, backtracking occurs probabilistically. This process generalizes the seminal Sinclair-Jerrum random walk (Sinclair & Jerrum, 1989) from the literature on approximate counting and sampling in theoretical computer science, and a conceptual contribution of our work is to highlight parallels with this literature. Empirically, we demonstrate on both synthetic and real language modeling tasks that VGB outperforms baselines on a variety of metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑