Terminal-RSI:通过递归自改写与策略优化实现的递归自改进终端智能体
Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine
AI总结:
该研究提出Terminal-RSI模型,通过交替进行递归自改写与策略优化,在多基准测试中显著提升终端智能体的任务解决能力与性能。
摘要:
将验证器结果转化为有效的策略更新,并将部分进展转化为完整的任务解决,对终端智能体至关重要。我们提出Terminal-RSI,其交替进行已验证轨迹的递归自改写(RSR)与继承评论家的递归策略优化(RPR)。我们提出Turn-Boundary Skip-Observation GAE(TB-GAE),以保留特定标记的基线,同时在交互回合间应用衰减;Family Dense Reward(FDR)校准不同任务系列间的语义进展;Success-Prioritized Reward(SPR)将严格成功与有界部分信用区分开。热身评论家模型、递归评论家交接以及阶段特定学习率协调两个优化阶段。从Qwen3.8-27B基础模型出发,经RSR-1、RPR-1、RSR-2和RPR-2的五阶段序列得到Terminal-RSI-27B:在Terminal-Bench 4.0上得19.84%,Terminal-Bench 2.1上得86.52%,Terminal-Bench 3.0上得16.95%,Terminal-Bench Hard上得79.00%;在LHTB上,过程奖励从0.21升至0.37,严格完成率从0/46升至5/46;SWR100成功率从3.00%翻倍至6.00%。通过将已验证轨迹改写与继承评论家的策略优化递归耦合,每个保留的阶段巩固发现的行为,逐步将智能体从局部部分修复推进至严格的终端验证。
英文摘要:
Converting verifier outcomes into useful policy updates and partial progress into complete task resolution is crucial for terminal agents. We present Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR). We formulate Turn-Boundary Skip-Observation GAE (TB-GAE) to retain token-specific baselines while applying decay across interaction turns. Family Dense Reward (FDR) calibrates semantic progress across task families; Success-Prioritized Reward (SPR) separates strict success from bounded partial credit. A Warm-up Critic Model, recursive critic handoff, and stage-specific learning rates coordinate the two refinement phases. The five-stage sequence from Qwen3.8-27B base model through RSR-1, RPR-1, RSR-2, and RPR-2 yields Terminal-RSI-27B: 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard. On LHTB, process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46; SWR100 success doubles from 3.00% to 6.00%. By recursively coupling verified trajectory rewriting with critic-inheriting policy refinement, each retained stage consolidates discovered behaviors and progressively advances the agent from partial local repairs to strict terminal verification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。