通过原始Wasserstein状态占用匹配实现从观察的离线模仿
Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching
AI总结：
本文提出PW-DICE方法，通过最小化原始Wasserstein状态占用距离并结合对比学习距离度量，统一了f-散度与Wasserstein最小化，在离线观察模仿中超越现有方法。
摘要：
在现实场景中，与环境的任意交互往往代价高昂，且专家示范的动作并不总是可用。为了同时减少对这两者的需求，离线从观察中学习（LfO）被广泛研究：智能体仅根据专家状态和与任务无关的非专家状态-动作对来学习解决任务。以SMODICE为代表的最先进的DIstribution Correction Estimation（DICE）方法，最小化学习者策略与经验专家策略之间的状态占用散度。然而，这类方法仅限于$f$-散度（KL和$\chi^2$）或使用Rubinstein对偶的Wasserstein距离，后者约束了底层距离度量，而该度量对基于Wasserstein的解决方案的性能至关重要。为了支持更灵活的距离度量，我们提出了Primal Wasserstein DICE（PW-DICE）。它最小化学习者与专家状态占用之间的原始Wasserstein距离，并利用对比学习得到的距离度量。理论上，我们的框架是SMODICE的推广，并且是第一个统一$f$-散度最小化与Wasserstein最小化的工作。实验上，我们发现PW-DICE在多个最先进方法上有所改进。代码可在https://github.com/KaiYan289/PW-DICE获取。
英文摘要：
In real-world scenarios, arbitrary interactions with the environment can often be costly, and actions of expert demonstrations are not always available. To reduce the need for both, offline Learning from Observations (LfO) is extensively studied: the agent learns to solve a task given only expert states and task-agnostic non-expert state-action pairs. The state-of-the-art DIstribution Correction Estimation (DICE) methods, as exemplified by SMODICE, minimize the state occupancy divergence between the learner's and empirical expert policies. However, such methods are limited to either $f$-divergences (KL and $chi^2$) or Wasserstein distance with Rubinstein duality, the latter of which constrains the underlying distance metric crucial to the performance of Wasserstein-based solutions. To enable more flexible distance metrics, we propose Primal Wasserstein DICE (PW-DICE). It minimizes the primal Wasserstein distance between the learner and expert state occupancies and leverages a contrastively learned distance metric. Theoretically, our framework is a generalization of SMODICE, and is the first work that unifies $f$-divergence and Wasserstein minimization. Empirically, we find that PW-DICE improves upon several state-of-the-art methods. The code is available at https://github.com/KaiYan289/PW-DICE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
