模仿约束下的离线多样性最大化
Offline Diversity Maximization Under Imitation Constraints
AI总结：
针对无监督技能发现方法需大量在线交互、难利用任务无关数据且缺技能效用量化指标的问题，提出带模仿约束的离线算法，结合Fenchel对偶等最大化互信息目标，在D4RL及12-DoF四足机器人数据集上验证有效且策略可迁移到真机。
摘要：
无监督技能发现领域近期取得了显著进展，该领域利用各类信息论目标作为多样性的衡量指标。尽管已有这些进展，挑战依然存在：现有方法需要大量在线交互，无法利用海量可用的任务无关数据，且通常缺乏技能效用的量化衡量标准。针对这些挑战，我们提出了一种用于无监督技能发现的原则性离线算法，该算法除了最大化多样性之外，还能确保每个习得技能在一定程度上模仿仅含状态的专家演示。我们的主要理论贡献是将Fenchel对偶、强化学习与无监督技能发现关联起来，以在KL散度状态占用约束下最大化互信息目标。此外，我们在标准离线基准D4RL以及从12自由度四足机器人采集的自定义离线数据集上验证了方法的有效性，在仿真中训练的策略可很好地迁移到真实机器人系统。
英文摘要：
There has been significant recent progress in the area of unsupervised skill discovery, utilizing various information-theoretic objectives as measures of diversity. Despite these advances, challenges remain: current methods require significant online interaction, fail to leverage vast amounts of available task-agnostic data and typically lack a quantitative measure of skill utility. We address these challenges by proposing a principled offline algorithm for unsupervised skill discovery that, in addition to maximizing diversity, ensures that each learned skill imitates state-only expert demonstrations to a certain degree. Our main analytical contribution is to connect Fenchel duality, reinforcement learning, and unsupervised skill discovery to maximize a mutual information objective subject to KL-divergence state occupancy constraints. Furthermore, we demonstrate the effectiveness of our method on the standard offline benchmark D4RL and on a custom offline dataset collected from a 12-DoF quadruped robot for which the policies trained in simulation transfer well to the real robotic system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
