arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.01769cs.SEcs.AI
收藏

Commit0:从零生成库

Commit0: Library Generation from Scratch

Cornell University(康奈尔大学);Cohere(Cohere公司)

Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gallé, Alexander M Rush

AI总结:

提出 Commit0 基准,要求 AI 智能体依据规范和交互式单元测试从零实现软件库,结果显示现有智能体均无法完整复现库,而交互反馈能显著提升测试通过率。

摘要:

为了对生成式系统进行超越专家级软件开发能力的基准测试,我们提出了 Commit0,这是一个要求 AI 智能体从零开始编写库的基准。智能体会获得一份概述该库 API 的规范文档以及一套交互式单元测试,目标是据此生成该 API 的实现。该实现通过运行这些单元测试来验证。作为基准,Commit0 旨在超越静态的一次性代码生成,转向要求智能体处理长篇自然语言规范、适应多阶段反馈,并生成具有复杂依赖关系的代码。Commit0 还提供了一个交互式环境,模型可在其中获得关于其所生成代码的静态分析和执行反馈。我们的实验表明,尽管当前智能体能够通过部分单元测试,但尚没有任何智能体能够完全复现完整的库。结果还表明,交互式反馈对于模型生成可通过更多单元测试的代码非常有用,这验证了促进此类反馈使用的基准的价值。

英文摘要:

With the goal of benchmarking generative systems beyond expert software development ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch. Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests, with the goal of producing an implementation of this API accordingly. The implementation is validated through running these unit tests. As a benchmark, Commit0 is designed to move beyond static one-shot code generation towards agents that must process long-form natural language specifications, adapt to multi-stage feedback, and generate code with complex dependencies. Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate. Our experiments demonstrate that while current agents can pass some unit tests, none can yet fully reproduce full libraries. Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate its use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑