交互式和混合模仿学习：可证明地超越行为克隆
Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
AI总结：
本文提出Warm Stagger方法，证明在按状态计算注释成本时，交互式和混合模仿学习可超越传统行为克隆方法。
摘要：
模仿学习（IL）是一种从专家那里学习序列决策制定策略的范式，利用离线演示、交互式注释或两者。最近的进展表明，当注释成本按轨迹计算时，仅依赖离线演示的行为克隆（BC）无法在一般情况下得到改进，这限制了交互式方法如DAgger的帮助条件。我们重新审视这一结论，并证明当注释成本按状态计算时，使用交互式注释的算法可以可证明地超越BC。具体而言：（1）我们证明了Stagger，一种每轮一个样本的DAgger变种，在低恢复成本设置下可证明地超越BC；（2）我们启动了混合IL的研究，其中智能体从离线演示和交互式注释中学习。我们提出了Warm Stagger，其学习保证并不比单独使用任一数据源更差。此外，受模仿学习实践中复合误差和冷启动问题的启发，我们给出一个MDP示例，其中Warm Stagger在注释成本上有显著优势；（3）在MuJoCo连续控制任务上的实验表明，当交互式和离线注释之间的成本比例适当时，交互式和混合方法始终优于BC。据我们所知，我们的工作是首次强调状态层面的交互式注释和混合反馈在模仿学习中的优势。
英文摘要：
Imitation learning (IL) is a paradigm for learning sequential decision making policies from experts, leveraging offline demonstrations, interactive annotations, or both. Recent advances show that when annotation cost is tallied per trajectory, Behavior Cloning (BC) which relies solely on offline demonstrations cannot be improved in general, leaving limited conditions for interactive methods such as DAgger to help. We revisit this conclusion and prove that when the annotation cost is measured per state, algorithms using interactive annotations can provably outperform BC. Specifically: (1) we show that Stagger, a one sample per round variant of DAgger, provably beats BC under low recovery cost settings; (2) we initiate the study of hybrid IL where the agent learns from offline demonstrations and interactive annotations. We propose Warm Stagger whose learning guarantee is not much worse than using either data source alone. Furthermore, motivated by compounding error and cold start problem in imitation learning practice, we give an MDP example in which Warm Stagger has significant better annotation cost; (3) experiments on MuJoCo continuous control tasks confirm that, with modest cost ratio between interactive and offline annotations, interactive and hybrid approaches consistently outperform BC. To the best of our knowledge, our work is the first to highlight the benefit of state wise interactive annotation and hybrid feedback in imitation learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
