LLMs have demonstrated significant potential in code generation tasks, achieving promising results at the function or statement level across various benchmarks. However, the complexities associated with creating code artifacts like classes, particularly within the context of real-world software repositories, remain underexplored. Prior research treats class-level generation as an isolated task, neglecting the intricate dependencies & interactions that characterize real-world software environments. To address this gap, we introduce RepoClassBench, a comprehensive benchmark designed to rigorously evaluate LLMs in generating complex, class-level code within real-world repositories. RepoClassBench includes "Natural Language to Class generation" tasks across Java, Python & C# from a selection of repositories. We ensure that each class in our dataset not only has cross-file dependencies within the repository but also includes corresponding test cases to verify its functionality. We find that current models struggle with the realistic challenges posed by our benchmark, primarily due to their limited exposure to relevant repository contexts. To address this shortcoming, we introduce Retrieve-Repotools-Reflect (RRR), a novel approach that equips LLMs with static analysis tools to iteratively navigate & reason about repository-level context in an agent-based framework. Our experiments demonstrate that RRR significantly outperforms existing baselines on RepoClassBench, showcasing its effectiveness across programming languages & under various settings. Our findings emphasize the critical need for code-generation benchmarks to incorporate repo-level dependencies to more accurately reflect the complexities of software development. Our work shows the benefits of leveraging specialized tools to enhance LLMs' understanding of repository context. We plan to make our dataset & evaluation harness public.
CodeMEM: AST-Guided Adaptive Memory for Repository-Level Iterative Code Generation
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervision
School of Cyber Science and Technology, University of Science and Technology of China(中国科学技术大学网络安全学院);Anhui Province Key Laboratory of Digital Security(安徽省数字安全重点实验室);The University of Hong Kong(香港大学)
DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving
University College London(伦敦大学学院);Sana Labs(Sana实验室);Uppsala University(乌普萨拉大学);Nanjing University(南京大学);Carnegie Mellon University(卡内基梅隆大学);KTH Royal Institute of Technology(瑞典皇家理工学院)
Co-Evolution of Types and Dependencies: Towards Repository-Level Type Inference for Python Code
Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所软件工程中心);Key Laboratory of System Software (Chinese Academy of Sciences), Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所系统软件重点实验室)
Beyond Function-Level Search: Repository-Aware Dual-Encoder Code Retrieval with Adversarial Verification
International Digital Economy Academy (IDEA)(国际数字经济学院);School of Electronic and Computer Engineering, Peking University(电子与计算机工程学院,北京大学);Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院,清华大学)
CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, Nanjing University of Science and Technology(南京理工大学计算机学院);School of Computer Science, Central China Normal University(中央财经大学计算机学院);School of Computing Technologies, RMIT University(皇家墨尔本理工学院计算技术学院);M3S Research Group, SEIS Unit, University of Oulu(奥卢大学M3S研究组,SEIS单元)
TDD-Agent: Test-Driven Reasoning for Code Generation
National College for Excellent Engineers, Beihang University(北京航空航天大学卓越工程师学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Qingdao Research Institute and Hangzhou Innovation Institute, Beihang University(北京航空航天大学青岛研究院与杭州创新研究院);School of Software, Beihang University(北京航空航天大学软件学院)
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学);ShanghaiTech University(上海科技大学);The University of Hong Kong(香港大学)
A Scalable Benchmark for Repository-Oriented Long-Horizon Conversational Context Management
State Key Laboratory of Complex & Critical Software Environment, School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院复杂与关键软件环境国家重点实验室)
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
Tencent(腾讯公司);Peking University(北京大学);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Zhejiang University(浙江大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Singapore Management University(新加坡国立大学)
Natural Language Summarization Enables Multi-Repository Bug Localization by LLMs in Microservice Architectures
Yildiz Technical University, Dept. of Computer Eng.(Yildiz技术大学,计算机工程系);Yildiz Technical University, Department of Computer Engineering(Yildiz技术大学,计算机工程系)