The instruction-following ability of Large Language Models (LLMs) has cultivated a class of LLM-based systems capable of approaching complex tasks such as making edits to large code repositories. Due to the high sensitivity and unpredictability of LLM behavior in response to changes in prompting, robust evaluation tools are needed to drive future iteration of these systems. We propose RES-Q, a natural language instruction-based benchmark for evaluating $\textbf{R}$epository $\textbf{E}$diting $\textbf{S}$ystems, which consists of 100 handcrafted repository editing tasks derived from real GitHub commits. Given an edit instruction and a code repository, RES-Q evaluates an LLM system's ability to interpret the instruction, navigate the repository to gather relevant information, and construct an appropriate edit that satisfies the specified criteria. We argue that evaluating LLMs in this way addresses issues with traditional benchmarks and provides a more holistic assessment of a model's abilities. We evaluate various state-of-the-art LLMs as language agents in a repository-editing system built on Qurrent OS, our language agent development software. Despite their 1% pass@1 performance difference on HumanEval, we find Claude Sonnet 3.5 outperforms GPT-4o by 12% pass@1 on RES-Q, indicating RES-Q's capacity to differentiate model capability as traditional benchmarks approach saturation. We further investigate token efficiency, performance relationships with existing benchmarks, and interesting disparities between closed and open-source LLMs. Code and dataset are available at https://github.com/Qurrent-AI/RES-Q.
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学);ShanghaiTech University(上海科技大学);The University of Hong Kong(香港大学)
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机学院,北京大学);Microsoft Research Asia(微软亚洲研究院)
SHERLOC: Structured Diagnostic Localization for Code Repair Agents
NVIDIA(英伟达);Santa Clara, CA 95051, USA(美国加利福尼亚州圣克拉拉);TU Darmstadt(达姆施塔特工业大学);National Research Center for Applied Cybersecurity ATHENE(国家应用网络安全研究中心 ATHENE)
National Engineering Research Center of Speech and Language Information Processing(语音与语言信息处理国家级工程研究中心);University of Science and Technology of China(中国科学技术大学);iFLYTEK Research(iFLYTEK研究院);University of California, Los Angeles(美国加州大学洛杉矶分校)
Pull Requests as a Training Signal for Repo-Level Code Editing
King's College London, UK(伦敦国王学院);Chinese University of Hong Kong, HK(香港中文大学);National University of Singapore, SG(新加坡国立大学);Microsoft Research Asia, CN(微软亚洲研究院);The Alan Turing Institute, UK(艾伦·图灵研究所)
Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(智能信息处理重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Agriculture Information Institute, CAAS(农业信息研究所,中国农业科学院)
Constraining Sequential Model Editing with Editing Anchor Compression
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China(中国科学技术大学语音及语言信息处理国家工程研究中心);Zhejiang University(浙江大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
CodeRAG: Finding Relevant and Necessary Knowledge for Retrieval-Augmented Repository-Level Code Completion
Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);Ant Group(蚂蚁集团)
SetupX: Can LLM Agents Learn from Past Failures in Functionality-Correct Code Repository Setup?
Shanghai Jiao Tong University(上海交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Independent Researcher(独立研究者);Jinan Inspur Data Technology Co., Ltd.(济南 Inspur 数据技术有限公司)