arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.14877cs.AI
收藏

超越结果反馈:基于轨迹质量的经验学习

Beyond Outcome Feedback: Trajectory-Quality-Guided Experience-based Learning

School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)

Bingjie Zhang, Yibo Yang, Qiantong Liu, Benshuo Fu, Yi Chang, Dandan Guo

AI总结:

针对现有基于经验的LLM智能体仅依赖结果反馈的局限,提出轨迹质量引导的经验学习框架TQG-Exp,引入轨迹质量损失与配对滚动执行优化经验,在三个基准上较基线实现显著性能提升。

摘要:

基于经验的学习使大型语言模型(LLM)智能体能够通过复用过往执行的知识实现性能随时间提升。然而现有方法通常依赖最终任务结果作为经验学习的反馈,忽略了执行过程,限制了其基于轨迹质量评估和优化经验的能力。为解决这一局限,我们提出TQG-Exp,即轨迹质量引导的经验学习框架,引入了过程反馈。具体而言,我们提出轨迹质量损失(TQL),该损失同时考虑任务完成情况和过程质量,提供超越最终任务结果的学习信号。基于该信号,我们采用带与不带检索经验的配对 rollout(滚动执行)来指导经验优化。我们根据轨迹的结果和轨迹质量选择代表性轨迹,以提取任务级和任务类型级的经验。同时,利用配对组之间的轨迹质量差异来估计检索经验的贡献并更新其效用。随后,将学习到的效用与语义相关性相结合,以指导后续的经验检索。在ALFWorld、WebShop和ScienceWorld三个基准上,使用两个骨干LLM的实验表明,TQG-Exp优于现有基于经验的方法,在DeepSeek-V4-Flash和Qwen3.5-Flash上,其Avg.@3指标分别比最强基线提升了4.50和8.55个百分点。

英文摘要:

Experience-based learning enables large language model (LLM) agents to improve over time by reusing knowledge from past executions. However, existing methods typically rely on final task outcomes as feedback for experience learning, overlooking the execution process and limiting their ability to evaluate and optimize experience based on trajectory quality. To address this limitation, we propose TQG-Exp, a trajectory-quality-guided experience-based learning framework that introduces process feedback. Specifically, we introduce Trajectory Quality Loss (TQL), which jointly considers task completion and process quality to provide a learning signal beyond final task outcomes. Based on this signal, we employ paired rollouts with and without retrieved experience to guide experience optimization. Representative trajectories are selected based on their outcomes and trajectory quality to extract task-level and task-type-level experiences. Meanwhile, differences in trajectory quality between the paired groups are used to estimate the contribution of retrieved experience and update its utility. The learned utility is then combined with semantic relevance to guide subsequent experience retrieval. Experiments on ALFWorld, WebShop, and ScienceWorld with two backbone LLMs show that TQG-Exp outperforms existing experience-based methods, improving Avg.@3 over the strongest baseline by 4.50 and 8.55 percentage points on DeepSeek-V4-Flash and Qwen3.5-Flash, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑