一种利用离线数据集从观测中进行模仿学习的对偶方法
A Dual Approach to Imitation Learning from Observations with Offline Datasets
AI总结：
提出DILO算法，通过直接学习多步效用函数量化动作对专家分布偏差的影响，利用对偶原理简化从观测模仿学习为演员-评论家学习，无需专家动作，可扩展至高维观测并提升性能。
摘要：
在难以设计奖励函数的环境中，示范是学习智能体任务规范的有效替代方案。然而，当机器人具有复杂、不直观的形态时，在智能体的动作空间中展示专家行为变得笨拙。我们考虑一个实际场景：智能体拥有一个与环境的先前交互数据集，并仅提供观测的专家示范。典型的从观测中学习方法需要学习逆动力学模型或判别器作为训练的中间步骤。这些中间单步模型中的误差会在下游策略学习或部署过程中累积。我们通过直接学习一个多步效用函数来克服这些限制，该函数量化每个动作如何影响智能体与专家访问分布的偏差。利用对偶原理，我们推导出DILO（从观测中进行对偶模仿学习），一种能够利用任意次优数据学习模仿策略而无需专家动作的算法。DILO将从观测中学习的问题简化为仅需学习一个演员和一个评论家，其复杂度与普通离线强化学习相似。这使得DILO能够优雅地扩展到高维观测，并在各个方面展示出改进的性能。项目页面（代码和视频）：https://hari-sikchi.github.io/dilo/
英文摘要：
Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function is difficult. However, demonstrating expert behavior in the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-only expert demonstrations. Typical learning from observations approaches have required either learning an inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's divergence from the expert's visitation distribution. Using the principle of duality, we derive DILO (Dual Imitation Learning from Observations), an algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board. Project page (code and videos): $\href{https://hari-sikchi.github.io/dilo/}{\text{hari-sikchi.github.io/dilo/}}$
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
