arXiv 2603.154012026-03cs.SEcs.AI

SWE-Skills-Bench：代理技能在真实软件工程中的实际帮助有多大？

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

MBZUAI（莫扎德人工智能大学）；Nanjing University（南京大学）；South China University of Technology（华南理工大学）；The University of New South Wales（新南威尔士大学）

Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, Lijie Hu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SWE-Skills-Bench，首个基于需求的基准，评估代理技能在真实软件工程中的边际效用，发现多数技能无明显提升，仅少数专业技能有帮助。

摘要：

代理技能，即在推理过程中注入的结构化过程知识包，越来越多用于增强LLM代理在软件工程任务中的表现。然而，其在端到端开发环境中的实际效用仍不明确。本文提出了SWE-Skills-Bench，首个基于需求的基准，通过将49个公开的软件工程技能与固定提交的GitHub仓库和具有明确接受标准的需求文档配对，生成约565个任务实例，覆盖六个软件工程子领域。我们引入了一个确定性验证框架，将每个任务的接受标准映射到基于执行的测试，实现有无技能的受控配对评估。结果表明，技能注入的收益远低于快速采用的预期：49个技能中有39个无通过率提升，平均收益仅为+1.2%。令牌开销从适度节省到增加451%，而通过率保持不变。只有7个专门技能产生有意义的提升（最高+30%），而三个技能因版本不匹配的指导与项目背景冲突而降低性能。这些发现表明，代理技能是一种狭窄的干预，其效用强烈依赖于领域适配、抽象级别和上下文兼容性。SWE-Skills-Bench为评估软件工程代理中技能的设计、选择和部署提供了测试平台。SWE-Skills-Bench可在https://github.com/GeniusHTX/SWE-Skills-Bench获取。

英文摘要：

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear, and no standard procedure tests an individual skill before installation. We present swe-skills-bench, a skill-centric, requirement-driven benchmark that measures how much individual agent skills improve performance on real-world software engineering (SWE) tasks. We start from 98 public SWE skills, each paired with a real GitHub repository pinned at a fixed commit. For each skill, we derive repository-specific requirements, create executable verifiers, and build a reproducible environment. Under each of four agent-model configurations, we run every task once without the skill and once with it, yielding 3,920 paired evaluations. For each requirement, we compute a paired delta, i.e., the change in outcome when the skill is added. We then aggregate these deltas across a skill's ten requirements into a per-skill utility profile, and use a statistical test to classify each skill as Golden (significantly helpful), Ineffective (no significant effect), or Harmful (significantly detrimental). Our results show that skills provide limited benefits: 92 of 98 skills have no statistically significant effect, and adding skills raises the average pass ratio only from 76.6% to 76.9%. The few helpful skills supply procedures that the agent would otherwise omit (gains of up to $+17.7$ points). In contrast, harmful skills anchor the agent on their own templates and override the task's explicit instructions (losses of up to -17.0 points). Our code is available at https://github.com/ControllableModel/SWE-Skills-Bench.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文