RepoMod-Bench: 一个通过实现无关测试进行代码仓库现代化的基准测试
RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing
AI总结:
RepoMod-Bench通过实现无关测试评估代码仓库现代化,揭示大规模自主现代化的挑战。
摘要:
人工智能编码代理的发展将前沿从简单的片段完成转移到了自主的仓库级工程。然而,在一般的代码仓库生成中,评估这些代理仍然存在困难,因为缺乏确定性的地面真实值导致指标模糊。通过自动化翻译进行的代码现代化提供了一种更严谨的替代方法,通过提供固定的地面真实值——源代码仓库;然而现有的基准测试仅限于小规模仓库,并且依赖于语言特定的单元测试,这些测试对代理可见,允许测试驱动的过拟合。我们通过引入一个基于实现无关评估范式的仓库级代码现代化基准测试框架来解决这些限制。该框架通过RepoMod-Bench实现:一个包含21个真实世界仓库的基准测试,具有标准化接口,涵盖8种编程语言。该基准测试包含160万行代码(LOC)和11,616个测试用例,仓库大小从14到211,000行代码不等。通过针对具有标准化接口的仓库,我们利用实现无关的测试套件来验证源代码和目标实现之间的功能等价性。这种黑盒方法确保了验证在不同语言之间保持一致,而我们的环境隐藏了所有测试套件,以防止测试驱动的捷径。评估四种最先进的代理配置显示出一个明显的缩放崩溃:在10,000行代码以下的项目上,平均通过率从91.3%下降到超过50,000行代码的项目上15.3%。这些结果表明,大规模的自主现代化仍然是一个重大的开放挑战。我们的基准测试和代码可在https://github.com/Modelcode-ai/mcode-benchmark上获得。
英文摘要:
The evolution of AI coding agents has shifted the frontier from simple snippet completion to autonomous repository-level engineering. However, evaluating these agents remains ill-posed in general code repository generation, where the lack of deterministic ground truth leads to ambiguous metrics. Code modernization via automated translation offers a more rigorous alternative by providing a fixed ground truth -- the source repository; yet existing benchmarks are limited to small-scale repositories and rely on language-specific unit tests visible to the agent, allowing test-driven overfitting. We address these limitations by introducing a benchmarking framework for repository-level code modernization built on an implementation-agnostic evaluation paradigm. This framework is instantiated through RepoMod-Bench: a benchmark of 21 real-world repositories with standardized interfaces, spanning 8 programming languages. The benchmark contains 1.6M lines of code (LOC) and 11,616 tests, with repository sizes ranging from 14 to 211K LOC. By targeting repositories with standardized interfaces, we utilize an implementation-agnostic test suite to verify functional equivalence between source and target implementations. This black-box approach ensures verification remains consistent across languages, and our environment hides all test suites from agents to prevent test-driven shortcuts. Evaluating four state-of-the-art agent configurations reveals a sharp scaling collapse: average pass rates drop from 91.3% on projects under 10K LOC to 15.3% on projects exceeding 50K LOC. These results demonstrate that autonomous modernization at scale remains a significant open challenge. Our benchmark and code are available at https://github.com/Modelcode-ai/mcode-benchmark.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。