EvoSkill v2:从基准适配到有状态演化器的全智能体技能演化系统
EvoSkill v2: Fully Agentic Skill Evolution from Benchmark Adaptation to a Stateful Evolver
AI总结:
EvoSkill v2是三智能体技能演化系统,通过有状态编码智能体演化技能,在两个基准上较基线提升保留通过率,且成本更低,还可让弱模型改进自身技能。
摘要:
EvoSkill v2是一个三智能体系统,用于编辑智能体所使用的指令(技能),而保留其模型和测试框架(harness)不变。用户通过描述一个基准并提供一个评分器(grader)来启动运行,基准适配器(benchmark adaptor)会将其转换为特定于该基准的插件。求解器(solver)尝试完成任务,演化器(evolver)读取求解器的训练轨迹(rollouts)并在每个轮次(epoch)重写技能。我们将演化器实现为一个带有bash工具的有状态编码智能体,用于整个运行过程,这与我们调研的技能优化器中两种最常见的设计不同:一种是像SkillOpt那样的一系列单模型调用,另一种是像SkillBoost那样每轮都重新启动的智能体。我们还测量了求解器每次尝试之间轨迹的差异,为演化器提供每个训练任务的多次尝试,并接受每一轮的技能,而非基于有噪声的分数在其中进行选择。在细粒度真实值的情况下,EvoSkill v2的技能在SpreadsheetBench v2上将保留通过率(held-out pass rate)提升至超过SkillOpt和SkillBoost中较强者的水平,幅度为7.7个百分点;在τ³-bench banking上则提升了5.4个百分点。在两个基准和两种反馈设置中,EvoSkill v2的演化器每提升一个百分点的保留通过率所花费的成本都低于任一基线优化器。EvoSkill v2还使能力较弱的模型能够改进自身技能,在τ³-bench banking实验中,当Claude Haiku 5.5同时作为求解器和演化器时,其提升幅度高达21.5个百分点。
英文摘要:
EvoSkill v2 is a three-agent system that edits the instructions (skills) an agent is given, leaving its model and harness intact. The user starts a run by describing a benchmark and supplying a grader, which a benchmark adaptor turns into a benchmark-specific plugin. A solver attempts the tasks, and an evolver reads the solver's training rollouts and rewrites the skills at each epoch. We implement the evolver as one stateful coding agent with a bash tool for the whole run, unlike the two most common designs among the skill optimizers we surveyed: a sequence of single model calls, as in SkillOpt, and an agent that starts afresh each round, as in SkillBoost. We also measure how much the solver's rollouts vary from one attempt to the next, give the evolver several attempts per training task, and accept every epoch's skills instead of selecting among them on noisy scores. With fine-grained ground truth, EvoSkill v2's skills raised the held-out pass rate above the stronger of SkillOpt and SkillBoost by 7.7 percentage points on SpreadsheetBench v2 and by 5.4 on $τ^3$-bench banking. In both benchmarks and both feedback settings, EvoSkill v2's evolver cost less per percentage point of held-out pass rate gained than either baseline's optimizer. EvoSkill v2 also enables a less capable model to improve its own skills, demonstrated by a gain of up to 21.5 percentage points in the $τ^3$-bench banking experiment where Claude Haiku 5.5 powers both solver and evolver.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。