arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2202.06003cs.LG
收藏

具有模型误设定的观测鲁棒学习

Robust Learning from Observation with Model Misspecification

EPFL(洛桑联邦理工学院);The Alan Turing Institute(阿兰·图灵研究所);The University of Edinburgh(爱丁堡大学);University of Cambridge(剑桥大学)

Luca Viano, Yu-Ting Huang, Parameswaran Kamalaruban, Craig Innes, Subramanian Ramamoorthy, Adrian Weller

摘要:

模仿学习是机器人系统中训练策略的一种流行范式,但在奖励函数难以确定的情况下,它的学习策略往往难以确定。然而,尽管IL算法的成功,他们强加的有点不切实际的要求,专家演示必须来自同一个域中,一个新的模仿者的政策是要学习。我们考虑一个实际的设置,其中(i)从真正的(部署)环境中的状态只专家演示给学习者,(ii)模仿学习者的政策是在模拟(训练)环境中训练的过渡动态是略有不同的真实环境,(iii)学习者没有任何访问的真实环境中的训练阶段超出了一批演示。目前的大多数IL方法,如生成对抗模仿学习及其仅状态变体,无法模仿上述设置下的最佳专家行为。通过利用鲁棒强化学习(RL)文献的见解,并建立在最近的对抗性模仿方法的基础上,我们提出了一个鲁棒的IL算法来学习可以有效地转移到真实环境中而无需微调的策略。此外,我们的经验表明,我们的方法优于国家的最先进的状态只有IL方法在真实环境中的zero-shot传输性能和鲁棒性能在不同的测试条件下。

英文摘要:

Imitation learning (IL) is a popular paradigm for training policies in robotic systems when specifying the reward function is difficult. However, despite the success of IL algorithms, they impose the somewhat unrealistic requirement that the expert demonstrations must come from the same domain in which a new imitator policy is to be learned. We consider a practical setting, where (i) state-only expert demonstrations from the real (deployment) environment are given to the learner, (ii) the imitation learner policy is trained in a simulation (training) environment whose transition dynamics is slightly different from the real environment, and (iii) the learner does not have any access to the real environment during the training phase beyond the batch of demonstrations given. Most of the current IL methods, such as generative adversarial imitation learning and its state-only variants, fail to imitate the optimal expert behavior under the above setting. By leveraging insights from the Robust reinforcement learning (RL) literature and building on recent adversarial imitation approaches, we propose a robust IL algorithm to learn policies that can effectively transfer to the real environment without fine-tuning. Furthermore, we empirically demonstrate on continuous-control benchmarks that our method outperforms the state-of-the-art state-only IL method in terms of the zero-shot transfer performance in the real environment and robust performance under different testing conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑