arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02502cs.CLcs.AIcs.LG
收藏

试错:面向LLM智能体的基于探索的轨迹优化

Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents

Allen Institute for AI(艾伦人工智能研究所);School of Computer Science, Peking University(北京大学计算机学院);UCLA(加利福尼亚大学洛杉矶分校);Ohio State University(俄亥俄州立大学);UIUC(伊利诺伊大学厄巴纳-香槟分校)

Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, Bill Yuchen Lin

AI总结:

该研究提出基于探索的轨迹优化方法ETO,让LLM智能体从探索失败中构建对比轨迹对,结合DPO等对比学习迭代更新策略,在三项复杂任务上性能大幅优于基线,且在无专家轨迹场景下具备潜力。

摘要:

大语言模型(LLMs)已成为各类自主智能体系统中不可或缺的组成部分。本研究提出一种基于探索的轨迹优化方法,称为ETO。该学习方法旨在提升开源LLM智能体的性能。与此前仅基于成功的专家轨迹进行训练的研究不同,我们的方法允许智能体从自身的探索失败中学习,通过迭代优化框架实现性能提升。在探索阶段,智能体在完成给定任务的过程中与环境交互,收集失败轨迹以构建对比轨迹对。在随后的训练阶段,智能体利用这些轨迹偏好对,通过DPO等对比学习方法更新其策略。这种探索与训练的迭代循环能够推动智能体持续进步。我们在三项复杂任务上开展的实验表明,ETO的性能始终大幅优于基线方法。此外,对任务求解效率以及在缺乏专家轨迹场景下潜力的分析,进一步验证了我们方法的有效性。

英文摘要:

Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO. This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑