UNIQ：用于避免不良示范的离线逆Q学习
UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations
AI总结：
针对离线学习中避免不良示范的问题，提出基于逆Q学习框架的UNIQ算法，通过最大化学习策略与不良策略间的统计距离实现非对抗性训练，并在基准环境中优于现有基线。
摘要：
我们解决了离线学习一种避免不良示范的策略的问题。传统的离线模仿学习方法旨在模仿专家或近最优示范，与此不同，我们的设定涉及避免不良行为（通过不良示范来指定）。为了解决这个问题，不同于以最小化学习策略与专家示范之间距离为目标的标准模仿学习，我们将学习任务表述为在状态-动作平稳分布空间中，最大化学习策略与不良策略之间的统计距离。这种截然不同的方法产生了一个新颖的训练目标，需要一种新算法来解决它。我们的算法UNIQ通过基于逆Q学习（inverse Q-learning）框架来解决这些挑战，将学习问题构建为一个合作性（非对抗性）任务。然后，我们展示了如何有效地利用无标签数据进行实际训练。我们的方法在标准基准环境中进行了评估，其表现始终优于最先进的基线方法。代码实现可在以下网址获取：https://github.com/hmh0/UNIQ。
英文摘要：
We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines. The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
