一种从观测和可能不完整轨迹的示例中进行离线模仿的简单解决方案
A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories
AI总结：
针对基于观测的离线模仿在不完整轨迹下 DICE 方法优化不稳定的问题，提出 TAILO，用判别器缩放的未来轨迹折扣和进行加权行为克隆，实验表明其更鲁棒有效。
摘要：
基于观测的离线模仿旨在解决这样的 MDP：其中只有任务特定的专家状态和任务无关的非专家状态-动作对可用。离线模仿在任意交互代价高昂且专家动作不可用的现实场景中非常有用。最先进的“DIstribution Correction Estimation”（DICE）方法最小化专家策略与学习者策略之间状态占用的散度，并通过加权行为克隆检索策略；然而，由于对偶域中的非鲁棒优化，当从不完整轨迹中学习时，其结果不稳定。为解决该问题，本文提出 Trajectory-Aware Imitation Learning from Observations（TAILO）。TAILO 使用沿未来轨迹的折扣和作为加权行为克隆的权重。该和的各项由判别器的输出缩放，判别器旨在识别专家状态。尽管简单，但如果任务无关数据中存在专家行为的轨迹或片段（先前工作中的常见假设），TAILO 就能良好工作。在跨多个测试平台的实验中，我们发现 TAILO 更加鲁棒且有效，尤其是在处理不完整轨迹时。
英文摘要：
Offline imitation from observations aims to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available. Offline imitation is useful in real-world scenarios where arbitrary interactions are costly and expert actions are unavailable. The state-of-the-art "DIstribution Correction Estimation" (DICE) methods minimize divergence of state occupancy between expert and learner policies and retrieve a policy with weighted behavior cloning; however, their results are unstable when learning from incomplete trajectories, due to a non-robust optimization in the dual domain. To address the issue, in this paper, we propose Trajectory-Aware Imitation Learning from Observations (TAILO). TAILO uses a discounted sum along the future trajectory as the weight for weighted behavior cloning. The terms for the sum are scaled by the output of a discriminator, which aims to identify expert states. Despite simplicity, TAILO works well if there exist trajectories or segments of expert behavior in the task-agnostic data, a common assumption in prior work. In experiments across multiple testbeds, we find TAILO to be more robust and effective, particularly with incomplete trajectories.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
