当示范遇到生成式世界模型：面向离线逆强化学习的最大似然框架
When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
AI总结：
针对离线IRL中世界模型误差会放大奖励估计偏差的问题，提出结合保守专家策略与最大似然的双层优化框架，并在MuJoCo和D4RL上显著超越现有基准。
摘要：
离线逆强化学习（Offline IRL）旨在从专家智能体的固定、有限示范集合中，恢复观测动作背后的奖励结构与环境动态。准确刻画执行任务时的专家行为，可应用于临床决策和自动驾驶等安全敏感场景。然而，观测动作中隐含的专家偏好结构，与专家对环境动态的模型（即“世界”模型）紧密相关。因此，在覆盖范围有限的有限数据上获得的不准确世界模型，可能会使奖励估计中的误差进一步累积。为解决该问题，我们将估计任务表述为一个双层优化问题：上层基于对专家策略（下层）的保守模型进行似然最大化。该策略模型的保守性体现在，它在最大化奖励时受到一个惩罚项约束，而该惩罚随所估计世界模型的不确定性增加而增大。我们提出了一个新的算法框架来求解该双层优化问题，并为相应的最优奖励估计量提供了统计与计算性能保证。最后，我们在 MuJoCo 的连续控制任务以及 D4RL 基准的不同数据集上证明，所提算法以显著优势优于最先进的离线 IRL 和模仿学习基准。
英文摘要：
Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the ``world'' model). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
