谁来完成工作？关于AI编码智能体拉取请求的后续修复与提交作者归属研究
Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests
浏览论文内容
中文总结 AI 辅助
本研究分析5个AI编码智能体的6774个合并PR，发现其后续修复几率是人类的1.62倍，但69.6%修复由原智能体完成，76.4%修复PR全提交为智能体撰写，表明智能体基本能自我完善。
中文摘要 AI 辅助
AI编码智能体现在在热门开源项目中合并的拉取请求（PR）中占据了很大份额。一个已合并的智能体PR通常被视为已完成的工作；然而，先前的研究报告了合并后智能体代码中存在的问题（例如，代码异味和静态分析问题）。然而，关于已合并的智能体PR在之后被修复的频率，以及实际执行修复的作者是谁，我们知之甚少。在本文中，我们追踪了来自AIDev-pop数据集（至少拥有500颗星的开源仓库）中五个AI编码智能体（OpenAI Codex、GitHub Copilot、Devin、Cursor和Claude Code）的6,774个已合并智能体PR，直至其后续修复，并以来自相同仓库的5,044个同期人类PR作为基线。我们将每个合并与其候选修复相关联，通过人类标注者和一个达到人类水平一致性的LLM评判者（二元Cohen's Kappa=0.78，而人类-人类K=0.77，直接修复精确度为90%）验证每个候选，并在PR和提交级别归因修复工作。我们的发现表明：（1）在相同时间段内，已合并的智能体PR在相同仓库中吸引已验证修复的几率是已合并人类PR的1.62倍；（2）智能体合并中69.6%的已验证修复来自同一智能体；（3）76.4%的已验证修复PR在所有提交中均由智能体撰写。这些结果表明，智能体目前在很大程度上完成了自己的工作，但它们的合并仍比人类合并更频繁地需要修复。
英文摘要
AI coding agents now author a large share of pull requests (PRs) merged into popular open-source projects. A merged agent PR is usually considered finished work; yet, prior studies have reported issues in agent code after the merge (e.g., code smells and static-analysis issues). However, little is known about how often a merged agent PR is fixed afterward, and who actually authors the fixing. In this paper, we follow 6,774 merged agent PRs across five AI coding agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) from the AIDev-pop dataset (open-source repositories with at least 500 stars) into their follow-up fixes, against a baseline of 5,044 contemporaneous human PRs from the same repositories. We link each merge to its candidate fixes, verify every candidate with human annotators and an LLM judge that matches human-level agreement (binary Cohen's Kappa=0.78 against a human-human K=0.77, Direct-fix precision 90%), and attribute the fixing work at the PR and the commit level. Our findings show that (1) merged agent PRs attract verified fixes at 1.62 times the odds of merged human PRs in the same repositories over the same period of time; (2) 69.6% of verified fixes in agent merges come from the same agent; and (3) 76.4% of the verified fix PRs are agent-authored throughout all commits. These results show that agents currently largely finish their own job, but their merges still require fixing more often than human merges.
发表机构
- Applied AI Initiatives (A2I2), Deakin University（迪肯大学）
机构由 AI 辅助整理，请以论文原文为准。
