arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.27136cs.AI
收藏

大型语言模型智能体的经验规则与策略联合学习

Joint Learning of Experiential Rules and Policies for Large Language Model Agents

Sun Yat-sen University(中山大学)

Shicheng Ye, Chao Yu

AI总结:

提出JERP框架,通过联合更新经验规则库和策略,使规则与演化策略对齐,在AlfWorld和WebShop上提升复杂交互任务的决策性能。

摘要:

对于多步交互环境中的LLM智能体,一个关键挑战是如何有效利用积累的交互经验。现有工作通常将这种经验分为两种用途:将其作为自然语言规则保留在模型外部以供后续提示,或使用轨迹和反馈更新模型参数。前者易于解释但可能随着策略演化而不同步;后者更广泛地改进策略,但在稀疏奖励设置中仅对局部错误提供有限的纠正。我们提出了LLM智能体的经验规则与策略联合学习(JERP),该方法从相同的交互轨迹中更新长期经验规则池和策略。在决策时,JERP检索任务相关规则,并将其与交互历史一起作为智能体的条件。每轮结束后,它使用收集的轨迹来优化策略,并通过将当前轨迹与参考成功轨迹进行比较来修订规则池。这种耦合使规则池与演化策略保持对齐,同时允许稳定有效的行为逐渐被吸收到模型本身中。在AlfWorld和WebShop上的实验表明,JERP在复杂交互任务的决策性能上取得了持续提升。

英文摘要:

For LLM agents in multi-step interactive environments, a key challenge is to make effective use of accumulated interaction experience. Existing work has typically separated two uses of such experience: keeping it outside the model as natural-language rules for later prompting, or using trajectories and feedback to update the model parameters. The former is easy to interpret but can fall out of sync with the evolving policy; the latter improves the policy more broadly but provides only limited correction for local mistakes in sparse-reward settings. We present Joint Learning of Experiential Rules and Policies for LLM Agents (JERP), which updates a long-term experiential-rule pool and the policy from the same interaction trajectories. At decision time, JERP retrieves task-relevant rules and conditions the agent on them together with the interaction history. After each episode, it uses the collected trajectories both to optimize the policy and to revise the rule pool by comparing current rollouts with reference successful trajectories. This coupling keeps the rule pool aligned with the evolving policy while allowing stable and effective behaviors to be gradually absorbed into the model itself. Experiments on AlfWorld and WebShop show that JERP yields consistent gains in decision performance for complex interactive tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑