How to evaluate Large Language Models (LLMs) in code generation is an open question. Existing benchmarks demonstrate poor alignment with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. This paper proposes a new benchmark - EvoCodeBench to address the preceding problems, which has three primary advances. (1) EvoCodeBench aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) EvoCodeBench offers comprehensive annotations (e.g., requirements, reference code, and reference dependencies), and robust evaluation metrics (e.g., Pass@k and Recall@k). (3) EvoCodeBench is an evolving benchmark to avoid data leakage. We build an automatic pipeline to update EvoCodeBench from the latest repositories. We release the first version - EvoCodeBench-2403, containing 275 samples from 25 real-world repositories. Based on EvoCodeBench, we propose repository-level code generation and evaluate 10 popular LLMs (e.g., gpt-4, gpt-3.5, DeepSeek Coder, StarCoder 2, CodeLLaMa, Gemma, and Qwen 1.5). Our experiments reveal the coding abilities of these LLMs in real-world repositories. For example, the highest Pass@1 of gpt-4 only is 20.73% in our experiments. We also analyze failed cases and summarize the shortcomings of existing LLMs in EvoCodeBench. We release EvoCodeBench, all prompts, and LLMs' completions for further community analysis.
EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems
Nanyang Technological University（南洋理工大学）；East China University of Science and Technology（东华大学）；Guangdong Ocean University（广东海洋大学）；City University of Hong Kong（香港城市大学）；Stanford University（斯坦福大学）
Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub
Tsinghua University（清华大学）；Renmin University of China（中国人民大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Shanghai Jiao Tong University（上海交通大学）
SetupX: Can LLM Agents Learn from Past Failures in Functionality-Correct Code Repository Setup?
Shanghai Jiao Tong University（上海交通大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Independent Researcher（独立研究者）；Jinan Inspur Data Technology Co., Ltd.（济南 Inspur 数据技术有限公司）
OpenSkill: Open-World Self-Evolution for LLM Agents
Lehigh University（莱维大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Salesforce AI Research（Salesforce人工智能研究）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）
