SkillGenBench: 评估LLM代理技能生成流水线的基准测试
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
AI总结：
本文提出SkillGenBench，一个用于评估LLM代理技能生成流水线的基准测试，通过统一可控的协议评估技能生成过程，涵盖任务条件生成和任务无关生成两种模式，以及基于仓库和文档的两种程序来源，揭示技能生成在不同数据源中的表现差异。
摘要：
随着LLM代理越来越多地围绕可重用的技能构建，一个核心挑战不再是代理是否能使用提供的技能，而是它们能否从仓库和文档中生成正确、可重用且可执行的技能。现有基准主要评估给定技能的有效性或代理从原始上下文中解决下游任务的能力，但并未将技能生成本身作为研究对象。我们引入SkillGenBench，一个用于评估技能生成流水线的基准测试，采用统一且受控的协议。在SkillGenBench中，生成器接收原始语料并生成标准化的技能制品，然后在固定框架下执行并经过统一的评估程序。该基准涵盖两种生成模式：任务条件生成，即在任务揭示后合成特定任务的技能；以及任务无关生成，即在下游任务确定前必须整理出可重用的技能库。它还涵盖两种互补的程序来源：基于仓库的实例，其中程序分布在代码、配置和脚本中；以及基于文档的实例，其中程序和约束必须从长文本中提炼。我们提供了标准化的任务规范、固定环境和以确定性执行为基础的评估协议，并辅以辅助信号用于诊断。在多种技能生成方法和基础模型上的实验显示了显著的性能差异，突显了可重用技能提炼的难度，并揭示了从软件仓库与长文本中生成技能的不同失败模式。SkillGenBench为研究技能生成作为代理系统中的独立研究问题建立了可重复的测试环境。
英文摘要：
As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
