代理技能在真实环境中的表现如何：在现实场景中评估LLM技能使用的基准测试
How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings
AI总结：
本文研究了在更现实的环境下LLM技能的实用性，发现技能效果随环境复杂性增加而下降，通过查询特定的技能细化策略可恢复性能，实验在Terminal-Bench 2.0上提升了Claude Opus 4.6的通过率。
摘要：
代理技能是一种可重用的领域特定知识工具，已成为扩展基于LLM的代理的流行机制，但正式评估技能使用性能仍十分有限。现有技能基准测试集中在过于理想化的条件，即LLM直接获得手工制作的、狭窄定制的任务特定技能，而许多现实场景中，LLM代理可能需要自行搜索和选择相关技能，甚至最接近匹配的技能可能也不适合任务。本文进行了首次全面研究，评估技能在渐进挑战性的现实环境中的实用性，其中代理必须从34000个现实技能中检索技能，且无法访问任何手工整理的技能。研究发现技能的益处是脆弱的：随着环境变得越来越现实，性能增益会持续下降，在最挑战性的场景中，通过率接近无技能基线。为了缩小这一差距，我们研究了技能细化策略，包括查询特定和查询无关的方法，证明查询特定的细化在初始技能相关且质量合理时能恢复丢失的性能。我们进一步展示了检索和细化在Terminal-Bench 2.0上的通用性，其中它们将Claude Opus 4.6的通过率从57.7%提高到65.5%。我们的结果在多个模型上一致，突显了技能在基于LLM的代理中的潜力和当前限制。我们的代码可在https://github.com/UCSB-NLP-Chang/Skill-Usage获取。
英文摘要：
Agent skills, which are reusable, domain-specific knowledge artifacts, have become a popular mechanism for extending LLM-based agents, yet formally benchmarking skill usage performance remains scarce. Existing skill benchmarking efforts focus on overly idealized conditions, where LLMs are directly provided with hand-crafted, narrowly-tailored task-specific skills for each task, whereas in many realistic settings, the LLM agent may have to search for and select relevant skills on its own, and even the closest matching skills may not be well-tailored for the task. In this paper, we conduct the first comprehensive study of skill utility under progressively challenging realistic settings, where agents must retrieve skills from a large collection of 34k real-world skills and may not have access to any hand-curated skills. Our findings reveal that the benefits of skills are fragile: performance gains degrade consistently as settings become more realistic, with pass rates approaching no-skill baselines in the most challenging scenarios. To narrow this gap, we study skill refinement strategies, including query-specific and query-agnostic approaches, and we show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality. We further demonstrate the generality of retrieval and refinement on Terminal-Bench 2.0, where they improve the pass rate of Claude Opus 4.6 from 57.7% to 65.5%. Our results, consistent across multiple models, highlight both the promise and the current limitations of skills for LLM-based agents. Our code is available at https://github.com/UCSB-NLP-Chang/Skill-Usage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
