arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.14895cs.LGcs.AI
收藏

通过在线专家纠正实现多轮LM代理的模仿学习

Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections

UC Berkeley(加州大学伯克利分校);Cornell University(康奈尔大学);Scale AI

Niklas Lauffer, Xiang Deng, Srivatsa Kundurthy, Brad Kenstler, Jeff Da

AI总结:

本文提出通过在线专家纠正解决多轮LM代理模仿学习中的协变量偏移问题,实验显示在软件工程任务中效果优于传统方法。

摘要:

训练LM代理的一种流行范式依赖于模仿学习,通过在专家轨迹上进行微调。然而,我们显示,对于多轮LM代理的模仿学习存在根本性的限制,即协变量偏移:当学生策略的行为偏离专家行为时,会遇到训练数据中不存在的状态,从而降低微调的有效性。受经典DAgger算法的启发,我们提出了一种新的数据生成方法,用于解决多轮LLM训练中的协变量偏移问题。我们引入了在线专家纠正(OECs),通过启动滚出时使用学生模型,然后在轨迹中途切换到专家模型来生成部分在线数据。我们探索了我们的数据生成技术在软件工程(SWE)任务中的有效性,这是一个多轮设置,其中LLM代理必须与开发环境互动以修复软件错误。我们的实验将OEC数据与各种其他在线和模仿学习方法在SWE代理问题上进行比较,并使用共同的拒绝采样(即使用环境奖励)结合监督微调技术进行训练。实验发现,OEC轨迹在7b和32b设置上分别比传统模仿学习提高了14%和13%,在SWE-bench验证中。我们的结果表明,结合专家演示和在线数据对于有效多轮LM代理训练是必要的。

英文摘要:

A popular paradigm for training LM agents relies on imitation learning, fine-tuning on expert trajectories. However, we show that the off-policy nature of imitation learning for multi-turn LM agents suffers from the fundamental limitation known as covariate shift: as the student policy's behavior diverges from the expert's, it encounters states not present in the training data, reducing the effectiveness of fine-tuning. Taking inspiration from the classic DAgger algorithm, we propose a novel data generation methodology for addressing covariate shift for multi-turn LLM training. We introduce on-policy expert corrections (OECs), partially on-policy data generated by starting rollouts with a student model and then switching to an expert model part way through the trajectory. We explore the effectiveness of our data generation technique in the domain of software engineering (SWE) tasks, a multi-turn setting where LLM agents must interact with a development environment to fix software bugs. Our experiments compare OEC data against various other on-policy and imitation learning approaches on SWE agent problems and train models using a common rejection sampling (i.e., using environment reward) combined with supervised fine-tuning technique. Experiments find that OEC trajectories show a relative 14% and 13% improvement over traditional imitation learning in the 7b and 32b setting, respectively, on SWE-bench verified. Our results demonstrate the need for combining expert demonstrations with on-policy data for effective multi-turn LM agent training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑