用 FIRE 对抗失败：在基于干预的学习中通过失败识别减轻专家负担
Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning
AI总结：
针对干预式模仿学习中专家监督负担重的问题，提出 FIRE，通过同时利用专家与非专家数据预测失败并触发纠正，显著提升样本效率与策略性能。
摘要：
监督式模仿学习（也称为行为克隆）会受到分布漂移的影响，从而导致策略执行过程中出现失败。缓解这一问题的一种方法是允许专家在任务执行期间纠正智能体的动作，其依据是专家判断智能体已经到达“不可返回点”。随后，使用这些新的纠正数据重新训练智能体的策略。仅靠这种方法可以学习到高性能智能体，但代价巨大：专家必须持续警觉地观察执行过程，直到策略达到指定的成功水平；即使到了那时，也无法保证策略总会成功。为解决这些局限，我们提出了 FIRE（Failure Identification to Reduce Expert Burden in intervention-based learning，即在基于干预的学习中通过失败识别减轻专家负担），这是一个能够预测正在运行的策略何时会失败、停止其执行并请求专家纠正的系统。与仅从专家数据中学习的现有方法不同，我们的方法同时从专家数据和非专家数据中学习，类似于对抗学习。我们在一系列具有挑战性的操作任务上通过实验证明，我们的方法能够识别导致失败的状态-动作对。这使其能够无缝集成到基于干预的学习系统中；在该系统中，我们表明，与最先进的逆强化学习方法相比，其样本效率提升了一个数量级，并且与使用等量数据通过行为克隆学习到的性能相比，性能得到显著提高。
英文摘要：
Supervised imitation learning, also known as behavioral cloning, suffers from distribution drift leading to failures during policy execution. One approach to mitigate this issue is to allow an expert to correct the agent's actions during task execution, based on the expert's determination that the agent has reached a `point of no return.' The agent's policy is then retrained using this new corrective data. This approach alone can enable high-performance agents to be learned, but at a substantial cost: the expert must vigilantly observe execution until the policy reaches a specified level of success, and even at that point, there is no guarantee that the policy will always succeed. To address these limitations, we present FIRE (Failure Identification to Reduce Expert Burden in intervention-based learning), a system that can predict when a running policy will fail, halt its execution, and request a correction from the expert. Unlike existing approaches that learn only from expert data, our approach learns from both expert and non-expert data, akin to adversarial learning. We demonstrate experimentally for a series of challenging manipulation tasks that our method is able to recognize state-action pairs that lead to failures. This permits seamless integration into an intervention-based learning system, where we show an order-of-magnitude gain in sample efficiency compared with a state-of-the-art inverse reinforcement learning method and dramatically improved performance over an equivalent amount of data learned with behavioral cloning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
