arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.00271cs.CL
收藏

数学推理中验证器引导搜索的扩展缺陷

Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Fei Yu, Yingru Li, Benyou Wang

AI总结:

本文揭示了数学推理中验证器引导搜索存在的扩展缺陷,即随样本量增加其优势递减并最终逊于重复采样,归因于不完美验证器导致的错误排序与剪枝,并探索了缓解方法。

摘要:

大型语言模型(LLMs)在多步推理上存在困难,推理时扩展已成为提升性能的一种有前景的策略。当样本数量受限时,验证器引导搜索通过选择并优先处理有效推理路径,其表现优于重复采样。然而,我们发现了一个关键局限:扩展缺陷,这在不同的模型(Mistral 7B和DeepSeekMath 7B)、基准(GSM8K和MATH)以及验证器(结果价值模型和过程奖励模型)中普遍存在。随着样本数量增加,验证器引导搜索的优势递减,最终表现不如重复采样。我们的分析将此归因于验证器失败,即不完美的验证器对候选对象错误排序,并错误地剪枝了所有有效路径。这些问题在具有挑战性和分布外问题中进一步加剧,限制了搜索有效性。为缓解验证器失败,我们探索减少对验证器的依赖,并使用两种简单方法进行了初步调查。我们的发现揭示了验证器引导搜索的根本局限,并提出了未来方向。

英文摘要:

Large language models (LLMs) struggle with multi-step reasoning, where inference-time scaling has emerged as a promising strategy for performance improvement. Verifier-guided search outperforms repeated sampling when sample size is limited by selecting and prioritizing valid reasoning paths. However, we identify a critical limitation: scaling flaws, prevalent across different models (Mistral 7B and DeepSeekMath 7B), benchmarks (GSM8K and MATH), and verifiers (outcome value models and process reward models). As sample size increases, verifier-guided search exhibits diminishing advantages and eventually underperforms repeated sampling. Our analysis attributes this to verifier failures, where imperfect verifiers misrank candidates and erroneously prune all valid paths. These issues are further exacerbated in challenging and out-of-distribution problems, restricting search effectiveness. To mitigate verifier failures, we explore reducing reliance on verifiers and conduct preliminary investigations using two simple methods. Our findings reveal fundamental limitations in verifier-guided search and suggest future directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑