利用验证-生成差距：基于置信度条件的测试时强化学习
Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
AI总结：
提出TTRL-CoCoV框架，通过置信度自适应机制解决无标签设置下Pass@k优化中的伪标签错误和多样性崩溃问题，显著提升Pass@1和Pass@k性能。
摘要：
测试时强化学习已成为一种有前景的范式，用于在完全无标签的方式下增强大型语言模型的复杂推理能力。尽管现有研究关注Pass@1性能，但在无标签设置下优化Pass@k（衡量生成覆盖率以支持持续探索）仍未被充分探索且至关重要。在无标签设置下优化Pass@k极具挑战性，因为直接应用对RLVR有效的Pass@k优势设计会导致性能不佳。通过深入的实证分析，我们发现阻碍性能的根本原因：低置信度样本的伪标签估计很可能不正确，而高置信度样本的候选答案则遭受严重的多样性崩溃。为克服这些障碍，我们提出TTRL-CoCoV（基于置信度条件的测试时强化学习），一种新颖的置信度自适应框架，可扩展Pass@k覆盖率并提升Pass@1性能。基于我们的关键洞察——验证能力通常领先于生成能力，TTRL-CoCoV采用置信度条件机制：对于高置信度样本，它引导验证器并应用探索增强奖励以防止多样性崩溃；对于低置信度样本，它将伪标签选择委托给验证器以过滤错误伪标签；对于中等置信度样本，则完全绕过验证。大量实验表明，TTRL-CoCoV在6个广泛认可的基准上优于最佳竞争方法，在Pass@1上平均绝对提升+9.8%，在Pass@16上平均绝对提升+18.7%，甚至在与全监督强化学习方法相比时，在多个推理基准上实现了高达+5.0%的Pass@1绝对提升。我们的代码仓库：此 https URL。
英文摘要：
Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. Through in-depth empirical analysis, we discover the root causes hindering performance: pseudo-label estimations for low-confidence samples have a high probability of being incorrect, while candidate answers for high-confidence samples suffer from severe diversity collapse. To overcome these hurdles, we propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 performance. Based on our key insight that verification capability generally leads generation capability, TTRL-CoCoV employs a confidence-conditioned mechanism: for high-confidence samples, it bootstraps verifier and applies an exploration-enhancing reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. Extensive experiments demonstrate that TTRL-CoCoV outperforms the best competing methods across 6 widely-recognized benchmarks, achieves average absolute gains of +9.8% in Pass@1 and +18.7% in Pass@16 over TTRL, and even achieves absolute Pass@1 improvements of up to +5.0% across multiple reasoning benchmarks when compared against fully supervised RL methods. Our code repository: https://github.com/shanjf666/CoCoV.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
