基于模型反向增强的离线模仿学习
Offline Imitation Learning with Model-based Reverse Augmentation
AI总结：
针对离线模仿学习中的协变量偏移问题，提出带自步反向增强的SRA框架，通过反向动力学模型生成增强轨迹并结合强化学习，有效缓解偏移并达到基准测试的SOTA性能。
摘要：
在离线模仿学习（IL）中，主要挑战之一是专家观测数据与智能体实际遇到的分布之间存在**协变量偏移**，因为当智能体处于专家演示的状态分布之外时，很难确定其应采取的动作。近期，无模型方法通过引入补充数据并识别与专家相似的潜在样本，在学习过程中扩充可靠样本；基于模型的方法则构建带保守性量化的前向动力学模型，随后在专家演示的邻域内生成额外轨迹。然而，在没有奖励监督的情况下，这些方法在专家支持范围外的区域往往过于保守，因为只有在接近专家观测状态的状态中，才存在能实现策略优化的优选动作。为了鼓励对专家未观测状态进行更多探索，我们提出了一种新颖的基于模型的框架，称为带自步反向增强的离线模仿学习（SRA）。具体而言，我们基于离线演示构建反向动力学模型，该模型能以自步的方式高效生成通往专家观测状态的轨迹；随后，我们使用后续的强化学习方法从增强后的轨迹中学习，实现从专家未观测状态到专家观测状态的转移。该框架不仅探索了专家未观测状态，还能引导在这些状态上最大化长期回报，最终实现超越专家数据的泛化能力。实验结果表明，我们的方法能有效缓解协变量偏移问题，并在离线模仿学习基准测试上取得了当前最优性能。项目主页：https://www.lamda.nju.edu.cn/shaojj/KDD24_SRA/。
英文摘要：
In offline Imitation Learning (IL), one of the main challenges is the \textit{covariate shift} between the expert observations and the actual distribution encountered by the agent, because it is difficult to determine what action an agent should take when outside the state distribution of the expert demonstrations. Recently, the model-free solutions introduce the supplementary data and identify the latent expert-similar samples to augment the reliable samples during learning. Model-based solutions build forward dynamic models with conservatism quantification and then generate additional trajectories in the neighborhood of expert demonstrations. However, without reward supervision, these methods are often over-conservative in the out-of-expert-support regions, because only in states close to expert-observed states can there be a preferred action enabling policy optimization. To encourage more exploration on expert-unobserved states, we propose a novel model-based framework, called offline Imitation Learning with Self-paced Reverse Augmentation (SRA). Specifically, we build a reverse dynamic model from the offline demonstrations, which can efficiently generate trajectories leading to the expert-observed states in a self-paced style. Then, we use the subsequent reinforcement learning method to learn from the augmented trajectories and transit from expert-unobserved states to expert-observed states. This framework not only explores the expert-unobserved states but also guides maximizing long-term returns on these states, ultimately enabling generalization beyond the expert data. Empirical results show that our proposal could effectively mitigate the covariate shift and achieve the state-of-the-art performance on the offline imitation learning benchmarks. Project website: \url{https://www.lamda.nju.edu.cn/shaojj/KDD24_SRA/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
