arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01286cs.SEcs.AIcs.CLcs.LG
收藏

BenchEvolver: 通过以解决方案为中心的进化进行前沿任务合成

BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

University of California, Berkeley(加州大学伯克利分校);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院)

Yangzhen Wu, Aaron J. Li, Wenjie Ma, Li Cao, Ziheng Zhou, Mert Cemri, Shu Liu, Yuran Xiu, Chenxiao Yan, Haikun Zhao, Bin Yu, Ion Stoica, Dawn Song

AI总结:

提出BenchEvolver框架,通过进化参考解决方案自动生成更难的编程问题,以解决基准饱和问题,并在LiveCodeBench和SciCode上验证其有效性。

摘要:

前沿大语言模型的快速进步导致了广泛的基准饱和,限制了现有数据集区分模型能力或提供有用训练信号的能力。例如,在LiveCodeBench上,前沿模型在简单拆分上达到超过99%的Pass@1,在不同难度级别上平均超过90%的Pass@1。构建新的、具有挑战性的数据集通常需要大量人力,成为进步的瓶颈。我们引入了BenchEvolver,一个以解决方案为中心的进化框架,自动将现有编码问题转化为更难的变体。BenchEvolver不是从头生成问题,而是通过结构化变换进化参考解决方案,并从进化后的解决方案中推导出相应的描述和测试。这种设计将生成过程基于可执行语义,使得能够可扩展地构建高质量、多样化和困难的任务,并具有可验证的正确性。将BenchEvolver应用于LiveCodeBench和SciCode,我们获得了显著更难的进化任务,同时保持了有效性、参考正确性和多样性。我们进一步策划了LiveCodeBench-Plus,一个包含91个问题的基准,结合了进化后的任务和困难的原始LCB-v6任务,其中前沿模型的Pass@1范围从27.5%到62.6%,恢复了强编码模型之间的清晰区分。重要的是,即使对于生成它们的模型,进化后的任务仍然具有挑战性,从而实现了自我改进。我们进一步表明,在进化后的LCB任务上进行强化学习提高了留出编码性能:对于gpt-oss-20b,种子+进化训练在LCB v6 Hard和LCB-Pro Easy上分别获得了+8.7和+8.3的Pass@1提升,分别超过仅种子训练的70.7%和34.8%。我们的结果表明,BenchEvolver可以将饱和的基准转化为前沿级别的评估套件和可重用的训练信号。

英文摘要:

The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑