Software engineering activities such as package migration, fixing errors reports from static analysis or testing, and adding type annotations or other specifications to a codebase, involve pervasively editing the entire repository of code. We formulate these activities as repository-level coding tasks.
Recent tools like GitHub Copilot, which are powered by Large Language Models (LLMs), have succeeded in offering high-quality solutions to localized coding problems. Repository-level coding tasks are more involved and cannot be solved directly using LLMs, since code within a repository is inter-dependent and the entire repository may be too large to fit into the prompt. We frame repository-level coding as a planning problem and present a task-agnostic framework, called CodePlan to solve it. CodePlan synthesizes a multi-step chain of edits (plan), where each step results in a call to an LLM on a code location with context derived from the entire repository, previous code changes and task-specific instructions. CodePlan is based on a novel combination of an incremental dependency analysis, a change may-impact analysis and an adaptive planning algorithm.
We evaluate the effectiveness of CodePlan on two repository-level tasks: package migration (C#) and temporal code edits (Python). Each task is evaluated on multiple code repositories, each of which requires inter-dependent changes to many files (between 2-97 files). Coding tasks of this level of complexity have not been automated using LLMs before. Our results show that CodePlan has better match with the ground truth compared to baselines. CodePlan is able to get 5/6 repositories to pass the validity checks (e.g., to build without errors and make correct code edits) whereas the baselines (without planning but with the same type of contextual information as CodePlan) cannot get any of the repositories to pass them.
VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);SUAT;Alibaba Token Hub, Alibaba Group(阿里巴巴集团阿里巴巴令牌中心);Alibaba Group(阿里巴巴集团)
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
Shanghai Jiao Tong University(上海交通大学);Xinjiang University(新疆大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Independent Researcher(独立研究者);The Chinese University of Hong Kong(香港中文大学)
Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学);Paul G. Allen School of Computer Science and Engineering(保罗·G·艾伦计算机科学与工程学院);Allen Institute of Artificial Intelligence(艾伦人工智能研究所);Machine Learning Department(机器学习系)
Merged, Not Measured: An Empirical Study of Performance Issues Fixed by Coding Agents
University of Arizona(亚利桑那大学);Wuhan University(武汉大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);California State University, Long Beach(加州州立大学长滩分校);Rochester Institute of Technology(罗切斯特理工学院)
Prometheus: Towards Long-Horizon Codebase Navigation for Repository-Level Problem Solving
University College London(伦敦大学学院);Sana Labs(Sana实验室);Uppsala University(乌普萨拉大学);Nanjing University(南京大学);Carnegie Mellon University(卡内基梅隆大学);KTH Royal Institute of Technology(瑞典皇家理工学院)
Shanghai Jiao Tong University(上海交通大学);Xi’an Jiaotong University(西安交通大学);Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术开发区(滨江)区块链与数据安全研究院)
A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents
Singapore Management University(新加坡管理大学);Monash University Malaysia(马来西亚莫纳什大学);Daegu Gyeongbuk Institute of Science and Technology (DGIST)(大邱庆北科学技术院);The Chinese University of Hong Kong(香港中文大学);North Carolina State University(北卡罗来纳州立大学);City University of Hong Kong(香港城市大学);The Hong Kong University of Science and Technology(香港科技大学);The University of Hong Kong(香港大学)