行为克隆就足够了吗？理解模仿学习中的时间跨度
Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
AI总结：
该研究从学习理论角度重新审视离线与在线模仿学习的差距，分析带对数损失的行为克隆，发现满足特定条件时离线IL可实现时间跨度无关的样本复杂度，且离线与在线IL的差距小于此前认知，实验验证了结果的实际相关性。
摘要：
模仿学习（IL）旨在通过从演示中学习，在序贯决策任务中模仿专家的行为，已广泛应用于机器人学、自动驾驶和自回归文本生成领域。模仿学习中最简单的方法——行为克隆（BC），被认为其样本复杂度与问题时间跨度呈不利的二次依赖关系，这促使了多种不同的在线算法的出现，这些算法在对数据和学习者访问专家的权限做出更强假设的情况下，实现了更优的线性时间跨度依赖。我们从学习理论的角度重新审视离线与在线模仿学习之间的明显差距，重点关注可实现/良好指定的设置，其中策略类包括深度神经网络在内的通用策略类。通过对带有对数损失的行为克隆进行新的分析，我们表明，只要满足以下两个条件，离线模仿学习就有可能实现与时间跨度无关的样本复杂度：（i）累积收益的范围受到控制；（ii）策略类的监督学习复杂度的适当概念受到控制。将我们的结果专门应用于确定性、平稳策略，我们发现离线与在线模仿学习之间的差距比之前认为的要小：（i）在密集奖励下，离线模仿学习有可能实现与时间跨度的线性依赖（与之前仅在在线模仿学习中可实现的结果相匹配）；（ii）在不对策略类做进一步假设的情况下，即使在良性马尔可夫决策过程（MDP）中，在线模仿学习也无法优于带有对数损失的离线模仿学习。我们通过在标准强化学习任务和自回归语言生成任务上的实验来补充理论结果，以验证我们发现的实际相关性。
英文摘要：
Imitation learning (IL) aims to mimic the behavior of an expert in a sequential decision making task by learning from demonstrations, and has been widely applied to robotics, autonomous driving, and autoregressive text generation. The simplest approach to IL, behavior cloning (BC), is thought to incur sample complexity with unfavorable quadratic dependence on the problem horizon, motivating a variety of different online algorithms that attain improved linear horizon dependence under stronger assumptions on the data and the learner's access to the expert. We revisit the apparent gap between offline and online IL from a learning-theoretic perspective, with a focus on the realizable/well-specified setting with general policy classes up to and including deep neural networks. Through a new analysis of behavior cloning with the logarithmic loss, we show that it is possible to achieve horizon-independent sample complexity in offline IL whenever (i) the range of the cumulative payoffs is controlled, and (ii) an appropriate notion of supervised learning complexity for the policy class is controlled. Specializing our results to deterministic, stationary policies, we show that the gap between offline and online IL is smaller than previously thought: (i) it is possible to achieve linear dependence on horizon in offline IL under dense rewards (matching what was previously only known to be achievable in online IL); and (ii) without further assumptions on the policy class, online IL cannot improve over offline IL with the logarithmic loss, even in benign MDPs. We complement our theoretical results with experiments on standard RL tasks and autoregressive language generation to validate the practical relevance of our findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
