提出 Commit0 基准,要求 AI 智能体依据规范和交互式单元测试从零实现软件库,结果显示现有智能体均无法完整复现库,而交互反馈能显著提升测试通过率。
摘要:
为了对生成式系统进行超越专家级软件开发能力的基准测试,我们提出了 Commit0,这是一个要求 AI 智能体从零开始编写库的基准。智能体会获得一份概述该库 API 的规范文档以及一套交互式单元测试,目标是据此生成该 API 的实现。该实现通过运行这些单元测试来验证。作为基准,Commit0 旨在超越静态的一次性代码生成,转向要求智能体处理长篇自然语言规范、适应多阶段反馈,并生成具有复杂依赖关系的代码。Commit0 还提供了一个交互式环境,模型可在其中获得关于其所生成代码的静态分析和执行反馈。我们的实验表明,尽管当前智能体能够通过部分单元测试,但尚没有任何智能体能够完全复现完整的库。结果还表明,交互式反馈对于模型生成可通过更多单元测试的代码非常有用,这验证了促进此类反馈使用的基准的价值。
英文摘要:
With the goal of benchmarking generative systems beyond expert software development ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch. Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests, with the goal of producing an implementation of this API accordingly. The implementation is validated through running these unit tests. As a benchmark, Commit0 is designed to move beyond static one-shot code generation towards agents that must process long-form natural language specifications, adapt to multi-stage feedback, and generate code with complex dependencies. Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate. Our experiments demonstrate that while current agents can pass some unit tests, none can yet fully reproduce full libraries. Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate its use.
CMAP, École Polytechnique, Institut Polytechnique de Paris(巴黎综合理工学院应用数学中心,巴黎理工学院);Samovar, Télécom SudParis, département CITI, TIPIC, Institut Polytechnique de Paris(萨莫瓦尔实验室,南巴黎电信学院,CITI系,TIPIC实验室,巴黎理工学院);Deep Mind(深度思维);Google Research, Brain Team(谷歌研究院大脑团队)
本文介绍了TRUNCATED REINFORCENT LEARNING FOR LANGUAGE(TrufLL),这是一种仅使用强化学习(RL)从头开始训练条件语言模型的新颖方法。由于RL方法不能成功地扩展到大的动作空间,我们使用通用的语言模型动态截断词汇空间。因此,TrufLL能够训练语言代理,只与其环境交互,没有任何特定于任务的先验知识;它只与任务无关的语言模型指导。有趣的是,这种方法避免了对标记数据集的依赖,并从本质上减少了预先训练的政策缺陷,如语言或暴露偏差。我们评估TrufLL的两个视觉问题生成任务,我们报告的性能和语言指标,这wethen证实了人类的评价积极的结果。据我们所知,这是第一种(几乎)从零开始成功学习语言生成策略的方法。
To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation
College of AI, Tsinghua University(清华大学人工智能学院);Proxseer Inc.(Proxseer公司);School of Computer Science, Nanjing University(南京大学计算机科学与技术学院);Software Institute, Nanjing University(南京大学软件学院);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
ELLIS Institute Tübingen(图宾根ELLIS研究所);MPI-IS Tübingen(图宾根MPI研究所);Mila Quebec AI Institute(魁北克AI研究所);Google(谷歌);Polytechnique Montréal(蒙特利尔高等理工学院);McGill University(麦吉尔大学);Moderne(Moderne公司);Gologic(Gologic公司);Tübingen AI Center(图宾根人工智能中心);Université de Montréal(蒙特利尔大学);ServiceNow Research(ServiceNow研究)
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Microsoft Research Asia(微软亚洲研究院);Korea University(高丽大学);University of Chinese Academy of Sciences(中国科学院大学);Microsoft Azure AI(微软Azure人工智能)
Understanding and Mitigating Library-Related Issues in LLM-Generated Code
IReSCoMath Research Laboratory, Faculty of Sciences, University of Gabes(加贝斯大学理学院 IReSCoMath 研究实验室);National School of Engineering, Gabes(加贝斯国立工程学院);Department of Electrical & Software Engineering, University of Calgary(卡尔加里大学电气与软件工程系)
COMET: Coverage-guided Model Generation For Deep Learning Library Testing
The Hong Kong University of Science and Technology(香港科技大学);Guangzhou HKUST Fok Ying Tung Research Institute(广州香港科技大学霍英东研究院);University of Waterloo(滑铁卢大学);Huazhong University of Science and Technology(华中科技大学)
针对现有 DL 模型生成技术层 API 覆盖不足的问题,本文提出 COMET,利用变异算子、覆盖引导搜索与模型综合提升测试多样性,并在八个 DL 库中发现 32 个新缺陷。
EZInput: A Cross-Environment Python Library for Easy UI Generation in Scientific Computing
Instituto de Tecnologia Química e Biológica António Xavier, Universidade Nova de Lisboa(里斯本新大学安东尼奥·阿瓦罗化学与生物理工学院);Faculty of Science and Engineering, Cell Biology, Åbo Akademi University(奥布学术院科学与工程学院细胞生物学系);InFLAMES Research Flagship Center, University of Turku(图卢大学InFLAMES旗舰研究中心);Turku Bioscience Centre, University of Turku and Åbo Akademi University(图卢大学生物科学中心(图卢大学与奥布学术院联合));Foundation for the Finnish Cancer Institute(芬兰癌症研究所基金会);UCL Laboratory for Molecular Cell Biology, University College London(伦敦大学学院分子细胞生物学实验室)