大规模评估智能体技能的框架
A Framework for Evaluating Agentic Skills at Scale
AI总结：
提出一个评估框架，通过构建真实任务和评分标准，大规模评估500个真实技能在19种智能体模型上的表现，发现模型对技能指令的遵循程度差异显著，且技能显著改变模型行为。
摘要：
智能体技能——结构化、可重用的知识工件，增强LLM智能体能力——已在工业界迅速采用，但其跨领域影响以及在商业和开源模型中的使用仍未得到充分研究，并且缺乏可复用的方法来评估单个技能。在这项工作中，我们提出了一个评估框架，允许技能作者构建真实任务，以严格评估技能中对他们最重要的方面，并通过解决这些任务来估计技能效用。此外，我们将评估方法大规模应用于500个真实技能，生成了1000个源自技能内容的任务，以及指令遵循和目标完成评分标准。使用这些指标，我们评估了19种智能体模型配置（包括专有和开源模型）在任务上的表现。我们的结果表明，模型在遵循技能中编码的指令方面差异很大，导致其性能提升存在显著差异。此外，我们表明，与无技能设置相比，访问技能显著改变了模型行为，为将主观工作流编码到LLM智能体中提供了一种重要机制。我们发布了评估数据集，以支持未来关于智能体技能的工作。
英文摘要：
Agent skills -- structured, reusable knowledge artifacts that augment LLM agent capabilities -- have been rapidly adopted in industry, yet their cross-domain impact and use across commercial and open-source models remain under-studied, and no reusable methodology exists for evaluating an individual skill. In this work, we present an evaluation framework that lets a skill author construct realistic tasks to rigorously assess the aspects of a skill that matter most to them, and that estimates skill utility by solving those tasks. Further, we apply our evaluation approach at scale to 500 real-world skills, generating 1,000 tasks derived from the skills' content, along with instruction-following and goal-completion scoring rubrics. Using these metrics, we evaluate how 19 agent-model configurations, both proprietary and open-source, perform on the tasks. Our results show that models vary widely in how closely they adhere to the instructions encoded in skills, leading to substantial differences in their performance gains. Furthermore, we show that access to a skill significantly changes model behavior compared to the no-skill setup, providing an essential mechanism for encoding opinionated workflows into LLM agents. We release our evaluation dataset to support future work on agent skills.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
