评分验证器：评估代码与推理领域的合成验证方法
Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
AI总结：
本研究提出将现有编码基准转换为评分排序数据集的方法，配套多维度评估指标，发布四个新基准，实验证实推理可提升测试用例生成效果、增加测试用例数能提高合成验证准确率。
摘要：
生成测试用例、奖励建模等合成验证技术，是提升大语言模型（LLM）编码能力、使其突破预定义测试限制的常用方法。此外，代码验证作为通过强化学习提升LLM推理能力的关键组件，近期已取得显著成效。本文提出一种方法，可将现有编码基准转换为评分与排序数据集，用于评估合成验证器的有效性。我们还提出多项指标，借助所构建的基准衡量合成验证器的不同维度性能。通过运用该方法，我们发布了四个新基准（HE-R、HE-R+、MBPP-R、MBPP-R+），并基于标准型、推理型与奖励型LLM对合成验证方法开展了分析。实验表明，推理能力可显著提升测试用例生成效果，且增加测试用例数量能提高验证准确率。
英文摘要：
Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verification has recently found great success as a critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose an approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-based, and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
