arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.26342cs.AIcs.CL
收藏

VibeTraining-Bench:用于迭代系统改进的数据-训练器-推理工具协同设计

VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement

Zhejiang University(浙江大学);Westlake University(西湖大学);Alibaba Group(阿里巴巴集团);South China University of Technology(华南理工大学);The Chinese University of Hong Kong(香港中文大学)

Ruotian Peng, Yuanyi Ren, Shuaiqi Liu, Xin Guo, Fang Zhu, Yangyi Huang, Yue Wu, Zheng Wang, Yandong Wen, Zhihang Fu

AI总结:

本文提出VibeTraining-Bench基准,用于评估智能体跨组件协同设计能力,实验显示前沿智能体可显著提升目标系统性能,该基准将推进递归自我改进研究。

摘要:

递归自我改进(RSI)要求智能体诊断系统瓶颈,并协调训练与推理环节的改进。我们将智能体联合优化训练数据、训练器及推理工具的能力称为跨组件协同设计。本文提出VibeTraining-Bench,这是一个用于评估该能力的基准,覆盖终端操作、软件工程、多轮工具使用及数学推理等任务。在每个任务中,研究智能体从相同的初始智能体系统出发,在固定资源预算下,通过验证反馈指导,迭代优化数据、训练器和推理工具,最终提交一个智能体系统在测试集上评估。为防止奖励作弊,我们设置三项保障措施:测试数据隔离、独立的研究与评分容器、研究智能体的网络限制。评估结果显示,前沿研究智能体可显著提升目标智能体系统性能,例如在SWE-bench Verified上,GPT-6-Astra将目标智能体系统的性能从32.9%提升至53.7%,与人类开发的指令模型的55.4%相当。不过,它们在协同设计及迭代优化数据、训练器和工具以实现累积性能提升方面仍有改进空间。我们希望VibeTraining-Bench能成为推进RSI、深化对智能体系统跨组件优化理解的统一平台。代码与数据可访问此https URL。

英文摘要:

Recursive self-improvement (RSI) requires agents to diagnose system bottlenecks and coordinate improvements across training and inference. We refer to the agents' capability to jointly optimize training data, the trainer, and the inference harness as cross-component co-design. We introduce VibeTraining-Bench, a benchmark for evaluating this capability across tasks spanning terminal operation, software engineering, multi-turn tool use, and mathematical reasoning. Within each task, research agents start from an identical initial agent system under a fixed resource budget. Guided by validation feedback, they iteratively refine data, trainers, and inference harnesses, then select and submit a final agent system for evaluation on the test set. To prevent reward hacking, we enforce three safeguards: test data isolation, separate research and grading containers, and network restrictions for research agents. Our evaluation results show that frontier research agents substantially improve target agent systems. For example, on SWE-bench Verified, GPT-6-Astra raises the target agent system's performance from 32.9% to 53.7%, comparable to the human-developed instruct model's 55.4%. However, there is still room to improve their ability to co-design and iteratively refine data, trainer, and harness to achieve cumulative performance gains. We hope that VibeTraining-Bench will serve as a unified platform for advancing RSI, deepening our understanding of cross-component optimization for agent systems. Code and data are available at https://github.com/D2I-ai/VibeTraining-Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑