arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.08232cs.LGcs.AI
收藏

用于逆强化学习的课程子目标

Curricular Subgoals for Inverse Reinforcement Learning

Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, Zhejiang University(浙江大学阿里巴巴-浙江大学前沿技术联合研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);College of Software Technology, Zhejiang University(浙江大学软件学院);Alibaba Group(阿里巴巴集团)

Shunyu Liu, Yunpeng Qing, Shuqi Xu, Hongyan Wu, Jiangtao Zhang, Jingyuan Cong, Tianhao Chen, Yunfu Liu, Mingli Song

AI总结:

针对现有逆强化学习全局奖励设计易受噪声干扰、在复杂多阶段任务中表现不佳的问题,提出基于课程子目标的CSIRL框架,通过动态选子目标、训练局部奖励生成器提升性能与可解释性,在D4RL和自动驾驶基准上优于SOTA方法。

摘要:

逆强化学习(Inverse Reinforcement Learning, IRL)旨在从专家演示中重构奖励函数以辅助策略学习,已在模仿学习中展现出显著成效。为实现类专家行为,现有IRL方法主要聚焦于学习全局奖励函数,以最小化模仿者与专家的轨迹差异。但这类全局设计仍受冗余噪声和误差传播问题的限制,会导致奖励分配不当,进而降低智能体在复杂多阶段任务中的能力。本文提出一种新颖的基于课程子目标的逆强化学习(Curricular Subgoal-based Inverse Reinforcement Learning, CSIRL)框架,该框架明确将单个任务拆解为若干局部子目标,以引导智能体模仿。具体而言,CSIRL首先引入已训练智能体在专家轨迹上的决策不确定性,动态选择子目标,这些子目标直接决定不同任务阶段的探索边界。为进一步获取各阶段的局部奖励函数,我们基于这些课程子目标定制了元模仿目标,用于训练内在奖励生成器。在D4RL和自动驾驶基准上的实验表明,所提方法的性能优于现有最优同类方法,且具备更好的可解释性。我们的代码可在https://github.com/Plankson/CSIRL获取。

英文摘要:

Inverse Reinforcement Learning (IRL) aims to reconstruct the reward function from expert demonstrations to facilitate policy learning, and has demonstrated its remarkable success in imitation learning. To promote expert-like behavior, existing IRL methods mainly focus on learning global reward functions to minimize the trajectory difference between the imitator and the expert. However, these global designs are still limited by the redundant noise and error propagation problems, leading to the unsuitable reward assignment and thus downgrading the agent capability in complex multi-stage tasks. In this paper, we propose a novel Curricular Subgoal-based Inverse Reinforcement Learning (CSIRL) framework, that explicitly disentangles one task with several local subgoals to guide agent imitation. Specifically, CSIRL firstly introduces decision uncertainty of the trained agent over expert trajectories to dynamically select subgoals, which directly determines the exploration boundary of different task stages. To further acquire local reward functions for each stage, we customize a meta-imitation objective based on these curricular subgoals to train an intrinsic reward generator. Experiments on the D4RL and autonomous driving benchmarks demonstrate that the proposed methods yields results superior to the state-of-the-art counterparts, as well as better interpretability. Our code is available at https://github.com/Plankson/CSIRL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑