软件工程（Software Engineering, SE）领域中的大语言模型（Large Language Models, LLMs）可为编程工作提供辅助。为严格评估LLM在实际编程场景中的能力，Carlos等人提出了SWE-bench数据集，该数据集包含从12个广泛使用的Python仓库中收集的2294个真实GitHub问题及其对应的拉取请求。近期已有多个性能优异的基于LLM的工具包在该数据集上完成开发与评估，但目前仍缺少针对SWE-bench数据质量的系统性评估。\n本文通过对SWE-bench数据集开展实证分析填补了这一空白。我们将研究期间位居SWE-bench排行榜首位的SWE-Agent+GPT-4作为评估对象，通过对比模型生成的补丁与实际拉取请求，对该模型成功解决问题的样本进行了人工筛查。\n分析揭示了SWE-bench数据集存在的若干关键问题：1）32.67%的成功补丁存在作弊行为，即问题报告或评论中直接提供了解决方案，我们将其称为解决方案泄露问题；2）31.08%的通过补丁属于可疑补丁，原因是测试用例过于薄弱，无法充分验证补丁的正确性。当我们过滤掉这些存在问题的样本后，SWE-Agent+GPT-4的问题解决率从12.47%降至3.97%。我们还发现，相同的数据质量问题也存在于SWE-bench的两个变体，即SWE-bench Lite和SWE-Bench Verified中。此外，超过94%的问题创建于LLM的知识截止日期之前，存在潜在的数据泄露风险。
英文摘要：
Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories. Several impressive LLM-based toolkits recently are developed and evaluated on this dataset. However, a systematic evaluation of the quality of SWE-bench remains missing. In this paper, we addressed this gap by presenting an empirical analysis of the SWE-bench dataset. We conducted a manual screening of instances where SWEAgent + GPT-4 successfully resolved issues by comparing the model-generated patches with the actual pull requests. SWE-Agent+GPT-4 was at the top of SWE-bench leaderboard during the time of our study. Our analysis reveals some critical issues with the SWE-bench dataset: 1) 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. We refer to as solution leakage problem. 2) 31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch. When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. We also observed that the same data quality issues also exist in the two variants of SWE-bench, i.e., SWE-bench Lite and SWE-Bench Verified. In addition, over 94% of the issues were created before LLM's knowledge cutoff dates, posing potential data leakage issues.
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
Shanghai Jiao Tong University（上海交通大学）；Xinjiang University（新疆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Independent Researcher（独立研究者）；The Chinese University of Hong Kong（香港中文大学）
SWE-Manager: Selecting and Synthesizing Golden Proposals Before Coding
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Science and Technology of China（中国科学技术大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
FPT Software AI Center（FPT软件人工智能中心）；School of Computing and Information Systems（计算与信息系统学院）；University of Melbourne（墨尔本大学）；Center of AI Research（人工智能研究中心）；VinUniversity（文大学）
SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents
The Chinese University of Hong Kong, China（香港中文大学）；Shanghai Jiao Tong University, China（上海交通大学）；Tsinghua University, China（清华大学）；ByteDance, China（字节跳动）
SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents
National University of Singapore（新加坡国立大学）；Vietnam National University, Hanoi（越南国家大学河内）；University of Virginia, USA（弗吉尼亚大学）；Agency for Science, Technology and Research (A*STAR), Singapore（新加坡科技研究局）；Tufts University, USA（塔夫茨大学）；Virginia Tech, USA（弗吉尼亚理工大学）；Salesforce AI Research, USA（Salesforce AI 研究）
SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes
Shanghai Jiao Tong University（上海交通大学）；North China Electric Power University（华北电力大学）；Tsinghua University（清华大学）；East China Normal University（华东师范大学）；Shanghai Innovation Institute（上海创新研究所）；Singapore Management University（新加坡管理大学）
SWE Context Bench: A Benchmark for Context Learning in Coding
Independent Researcher（独立研究员）；University of Oxford（牛津大学）；University of Edinburgh（爱丁堡大学）；Technical University of Munich（慕尼黑工业大学）；MemoraX AI；National University of Singapore（新加坡国立大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
The Chinese University of Hong Kong（香港中文大学）；Independent（独立）；ELLIS；Technical University of Darmstadt（达姆施塔特技术大学）；Johns Hopkins University（约翰霍普金斯大学）；Monash University（墨尔本大学）
Lero, the Research Ireland Centre for Software, University of Limerick（利默里克大学 Lero 爱尔兰软件研究中心）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；University of Ottawa（渥太华大学）
SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
University of Toronto（多伦多大学）；Xiaohongshu Inc.（小红书）；Coolwei AI Lab（酷维人工智能实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加利福尼亚大学伯克利分校）
ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance
NVIDIA（英伟达）；University College London（伦敦大学学院）；New York University（纽约大学）；Georgia Institute of Technology（佐治亚理工学院）；Cornell University（康奈尔大学）；Stanford University（斯坦福大学）；Northeastern University（东北大学）
A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents
Singapore Management University（新加坡管理大学）；Monash University Malaysia（马来西亚莫纳什大学）；Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院）；The Chinese University of Hong Kong（香港中文大学）；North Carolina State University（北卡罗来纳州立大学）；City University of Hong Kong（香港城市大学）；The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）
