通过最优传输的测试时验证:覆盖率、ROC与次优性
Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality
AI总结:
该研究将带验证的测试时扩展建模为最优传输问题,统一刻画覆盖率、验证器收敛域与采样次优性的相互作用机制,提出两类采样算法并经多模型实验验证理论结论。
摘要:
尽管带验证的测试时扩展在提升大语言模型(LLMs)性能方面已展现出前景,但验证器的作用及其缺陷仍未得到充分探究。验证的效果通过三个量的相互作用体现:(i)生成器的覆盖率,(ii)验证器的收敛域(ROC),以及(iii)采样算法的次优性。尽管近期研究涵盖了这些因素的子集,但目前仍缺少一个量化其相互作用几何特性的统一框架。我们将可验证的测试时扩展建模为一个传输问题。这一框架刻画了覆盖率、ROC与次优性之间的相互作用,并揭示次优性-覆盖率曲线呈现三种机制:传输机制——其中次优性随覆盖率上升而增加;策略改进机制——其中次优性可能随覆盖率上升而降低,具体取决于验证器的ROC;饱和机制——其中次优性进入平台期,不受覆盖率影响。我们进一步提出并分析了两类采样算法——序贯式与批处理式,并考察了它们的计算复杂度如何影响这些权衡关系。在Qwen、Llama和Gemma模型上的实验结果证实了我们的理论发现。
英文摘要:
While test-time scaling with verification has shown promise in improving the performance of large language models (LLMs), the role of the verifier and its imperfections remain underexplored. The effect of verification manifests through interactions of three quantities: (i) the generator's coverage, (ii) the verifier's region of convergence (ROC), and (iii) the sampling algorithm's sub-optimality. Though recent studies capture subsets of these factors, a unified framework quantifying the geometry of their interplay is missing. We frame verifiable test-time scaling as a transport problem. This characterizes the interaction of coverage, ROC, and sub-optimality, and uncovers that the sub-optimality--coverage curve exhibits three regimes. A transport regime -- where sub-optimality increases with coverage, a policy improvement regime -- where sub-optimality may decrease with coverage, depending on the verifier's ROC, and a saturation regime -- where sub-optimality plateaus, unaffected by coverage. We further propose and analyze two classes of sampling algorithms -- sequential and batched, and examine how their computational complexities shape these trade-offs. Empirical results with Qwen, Llama, and Gemma models corroborate our theoretical findings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。