SPRINQL：由次优演示驱动的离线模仿学习
SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
AI总结：
本文提出基于逆向soft-Q学习的离线模仿学习方法SPRINQL，通过赋予专家演示更高权重、次优演示较低权重，将问题转化为Q函数空间上的凸优化，有效利用次优演示并避免过拟合，在离线IL基准上达到SOTA性能。
摘要：
我们关注离线模仿学习（IL），其旨在无需与环境交互的情况下，利用演示来模仿专家的行为。离线 IL 中的一个主要挑战是专家演示的有限支持，这些演示通常仅覆盖状态-动作空间的一小部分。虽然获取大量专家演示可能不可行，但通常可以收集到更大规模的次优演示集。例如，在治疗优化问题中，针对不同的慢性病有不同级别的医生治疗方案。这些方案涵盖从治疗专家和经验丰富的全科医生到经验不足的全科医生。类似地，当训练机器人在日常任务中模仿人类时，它们可以从具有不同专业水平和效率的个体中学习。在本文中，我们提出了一种离线 IL 方法，该方法在有效模仿专家轨迹的同时，利用了更大规模的次优演示集。现有的基于行为克隆或分布匹配的离线 IL 方法通常面临诸如对有限的专家演示集过拟合，或无意中从更大规模数据集中模仿次优轨迹等问题。我们的方法基于逆向 soft-Q 学习，同时从专家和次优演示中学习。它（通过学习到的权重）赋予与专家演示对齐更高的重要性，而赋予与次优演示对齐较低的重要性。我们方法（称为 SPRINQL）的一个关键贡献是将离线 IL 问题转化为 Q 函数空间上的凸优化。通过全面的实验评估，我们证明 SPRINQL 算法在离线 IL 基准上取得了最先进的（SOTA）性能。代码可在 https://github.com/hmhuy0/SPRINQL 获取。
英文摘要：
We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
