EvoTrainer: 协同进化LLM策略与训练框架以实现自主智能体强化学习
EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning
AI总结：
提出EvoTrainer框架，通过协同进化LLM策略和训练端框架，基于经验反馈自动诊断、修正并积累可复用技能，在数学推理、编程竞赛和仓库级软件工程任务上匹配或超越人工设计的RL基线。
摘要：
自主LLM训练通常被表述为配方搜索，这使训练框架基本保持静态。这种局限性在智能体RL中尤为突出，其中不断变化的瓶颈和标量奖励掩盖了多种失败模式。我们引入了EvoTrainer，一个通过经验反馈协同进化LLM策略和训练端框架的自主训练框架：它诊断rollout级别的证据、修正诊断、回测干预并积累可复用技能。在数学推理、竞赛编程代码生成和仓库级软件工程上的评估表明，在相同数据、代码库和评估协议下，EvoTrainer匹配或超过了人工设计的RL参考，其中在长周期智能体SWE上增益最大。轨迹分析显示，保留的策略在不同领域分化，进化的诊断阻止了无效的高分分支被提升，而可复用技能塑造了后续搜索。自主LLM RL应超越配方搜索，转向策略和解释它们的训练框架的联合进化。
英文摘要：
Autonomous LLM training is often framed as recipe search, which leaves the training harness largely static. This limitation sharpens in agentic RL, where shifting bottlenecks and scalar rewards mask diverse failure modes. We introduce EvoTrainer, an autonomous training framework that co-evolves LLM policies and training-side harnesses through empirical feedback: it diagnoses rollout-level evidence, revises diagnostics, backtests interventions, and accumulates reusable skills. Evaluated on mathematical reasoning, competitive-programming code generation, and repository-level software engineering, EvoTrainer matches or exceeds the human-engineered RL references under the same data, codebase, and evaluation protocol, with the largest gain on long-horizon agentic SWE. Trajectory analyses show that retained strategies diverge across domains, evolving diagnostics prevent invalid high-scoring branches from being promoted, and reusable skills shape later search. Autonomous LLM RL should move beyond recipe search toward joint evolution of policies and the training harnesses that interpret them.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
