arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节（Veterans Day），12 日无更新
arXiv 2501.13699cs.CLcs.SE
收藏

DI-BENCH：基于可测试仓库的大规模依赖推断大语言模型基准

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

Microsoft（微软公司）；Wuhan University（武汉大学）；Tongji University（同济大学）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）
Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie, Maoquan Wang, Yufan Huang, Elsie Nallipogu, Qingwei Lin, Yi… 展开作者

Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie, Maoquan Wang, Yufan Huang, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

DI-BENCH是一个大规模基准，用于评估大语言模型在仓库依赖推断上的能力，包含581个仓库，最佳模型执行通过率仅42.9%，凸显改进空间。

摘要：

大型语言模型已推进了自动化软件开发，然而，正确推断依赖关系仍然是一个挑战，即识别仓库成功运行所需的内部组件和外部包。现有研究强调，与依赖相关的问题导致生成仓库中超过40%的观察到的运行时错误。为解决此问题，我们引入了DI-BENCH，一个专门设计用于评估LLMs在依赖推断能力上的大规模基准和评估框架。该基准包含581个仓库，涵盖Python、C#、Rust和JavaScript的测试环境。基于文本和执行指标的广泛实验表明，当前性能最佳的模型仅达到42.9%的执行通过率，表明仍有显著改进空间。DI-BENCH为评估LLMs在仓库上的性能建立了新视角，为更稳健的端到端软件合成铺平了道路。

英文摘要：

Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑