arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.02176cs.CL
收藏

在生态系统层面组织、协调和基准测试代理技能

Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, Shuyue Hu

AI总结:

本文提出AgentSkillOS框架,通过树状检索和DAG协调提升大规模技能生态系统的管理和任务解决能力。

摘要:

Claude代理技能的迅速普及引发了如何有效利用、管理和扩展代理技能生态系统的根本问题。在本文中,我们提出了AgentSkillOS,这是第一个系统性的框架,用于技能选择、协调和生态系统层面的管理。AgentSkillOS包含两个阶段:(i)管理技能,通过节点级递归分类将技能组织成能力树,以实现高效的发现;和(ii)解决任务,通过基于DAG的管道检索、协调和执行多个技能。为了评估代理调用技能的能力,我们构建了一个包含30个富内容任务的基准测试,涵盖五个类别:数据计算、文档创建、运动视频、视觉设计和网络交互。我们使用基于LLM的成对评估来评估任务输出的质量,并通过Bradley-Terry模型汇总结果以产生统一的质量评分。在三个不同规模的技能生态系统(200到200,000个技能)上的实验表明,基于树的检索能有效近似于Oracle技能选择,而基于DAG的协调即使在给定相同技能集的情况下也显著优于原生的扁平调用。我们的发现确认了结构化组合是解锁技能潜力的关键。我们的GitHub仓库地址为:https://github.com/ynulihao/AgentSkillOS。

英文摘要:

The rapid proliferation of Claude agent skills has raised the central question of how to effectively leverage, manage, and scale the agent skill ecosystem. In this paper, we propose AgentSkillOS, the first principled framework for skill selection, orchestration, and ecosystem-level management. AgentSkillOS comprises two stages: (i) Manage Skills, which organizes skills into a capability tree via node-level recursive categorization for efficient discovery; and (ii) Solve Tasks, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines. To evaluate the agent's ability to invoke skills, we construct a benchmark of 30 artifact-rich tasks across five categories: data computation, document creation, motion video, visual design, and web interaction. We assess the quality of task outputs using LLM-based pairwise evaluation, and the results are aggregated via a Bradley-Terry model to produce unified quality scores. Experiments across three skill ecosystem scales (200 to 200K skills) show that tree-based retrieval effectively approximates oracle skill selection, and that DAG-based orchestration substantially outperforms native flat invocation even when given the identical skill set. Our findings confirm that structured composition is the key to unlocking skill potential. Our GitHub repository is available at:https://github.com/ynulihao/AgentSkillOS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑