发表机构
Peking University（北京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究构建RepoComplianceBench基准测试编码智能体对开源社区AI贡献规则的合规性，发现当前智能体几乎不主动检索规则，仅在提示等辅助下实现披露与验证，但始终不执行AI禁令。
AI 中文摘要
开源社区中充斥着AI生成的贡献，为应对这一情况，各社区已制定贡献规则来规范编码智能体的行为，涵盖全面禁止、强制披露、验证关卡及人工审核等类型。但目前编码智能体是否会阅读并遵循这些规则，以及在开源仓库中的实际表现仍不明确。为评估编码智能体在现实场景中的规则合规性，我们从49个包含AI贡献规则的仓库中整理出106个问题，构建了RepoComplianceBench基准。我们根据各仓库的规则判断每次运行的轨迹，测量智能体是否拒绝贡献、如实披露其辅助情况、通过所需的验证关卡，或将关键步骤上报给人工。我们还测试了额外提示、规则披露或合规验证者的反馈是否对合规性有帮助。我们在四个前沿模型上开展的实验显示，当前的智能体几乎不会主动检索贡献规则；智能体可通过提醒提示、规则引用及验证者反馈来实现披露和通过验证，但在我们测试的所有条件下，它们从未在禁止AI贡献的仓库中拒绝贡献。这一现状表明，验证和披露问题可通过现有机制解决，但执行禁令及人工上报仍是未解决的问题。
英文摘要
Open source communities have been flooded with AI-generated contributions. In defense, they have written contribution rules to regulate coding agents' behavior, spanning from a total ban, mandatory disclosure, to verification gates and human sign-offs. Yet, whether coding agents read and follow those rules, and behave in open source repositories, remains unknown. To estimate real-world rule compliance of coding agents, we curate 106 issues from 49 repositories containing AI contribution rules into RepoComplianceBench. We judge the trajectory of each run against the repository's rules, measuring whether the agent refuses to contribute, discloses its assistance truthfully, clears the required verification gates, or escalates critical steps to a human. We also test if extra prompts, rule disclosure, or feedback from the compliance verifier help with the situation. Our experiments on four frontier models show that today's agents almost never proactively retrieve the contribution rules. Agents pick up disclosure and verification with reminder prompts, rule quotes, and verifier feedback; however, they never refuse to contribute in AI-banned repositories under any condition we tested. The status reveals that verification and disclosure issues are solvable with existing mechanisms, yet enforcing bans and human escalations remains an open problem.
