反事实行为克隆：从不完美的人类演示中进行离线模仿学习
Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrations
AI总结：
本研究提出Counter-BC算法，通过扩展数据集纳入反事实动作，从不完美人类演示中恢复潜在策略，实现更高效的离线模仿学习，经实验验证其性能优于现有方法。
摘要：
从人类处学习具有挑战性，因为人类并非完美的教师。当普通人向机器人展示其希望机器人执行的新任务时，人类不可避免会犯错（例如输入带噪声的动作）并提供次优示例（例如超出目标）。现有方法通常通过匹配人类部分或全部行为进行学习，但该方法存在根本局限，因为演示本身并不完美。本研究通过让机器人在邻近动作间进行外推，而非仅考虑人类实际展示的动作，推进离线模仿学习。我们假设所有人类演示均试图传达潜在策略，而人类行为中的噪声和次优性会混淆数据并引入非预期复杂性。为恢复潜在策略并学习人类教师的意图，我们提出Counter-BC，这是行为克隆的广义版本。Counter-BC将数据集扩展为包含与人类演示动作相近的动作（即人类教师本可能意图但未实际展示的反事实动作）。训练期间，Counter-BC在该扩展区域内自主修改人类演示，以得到简化策略，该策略可解释人类数据中的潜在趋势。理论上，我们证明Counter-BC可从不完美数据、多个用户及不同技能水平的教师中提取与演示相似的简单策略。实验上，我们在带噪声的演示、标准化数据集及真实人类教师的设置中，将Counter-BC与最先进的替代方法进行比较。总体而言，我们发现通过考虑邻近动作来外推人类教师意图，可实现更熟练的人类学习。可在此处查看我们工作的视频：this https URL
英文摘要：
Learning from humans is challenging because people are imperfect teachers. When everyday humans show the robot a new task they want it to perform, humans inevitably make errors (e.g., inputting noisy actions) and provide suboptimal examples (e.g., overshooting the goal). Existing methods often learn by matching some or all of the human's behavior --- but this approach is fundamentally limited because the demonstrations themselves are imperfect. In this work we advance offline imitation learning by enabling robots to extrapolate across nearby actions, instead of only considering what the human actually showed. We achieve this by hypothesizing that all of the human's demonstrations are trying to convey an underlying policy, while the noise and sub-optimality within their behaviors obfuscates the data and introduces unintentional complexity. To recover the underlying policy and learn what the human teacher meant, we introduce Counter-BC, a generalized version of behavior cloning. Counter-BC expands the dataset to include actions close to behaviors the human demonstrated (i.e., counterfactual actions that the human teacher could have intended, but did not actually show). During training Counter-BC autonomously modifies the human's demonstrations within this expanded region to reach a simplified policy that explains the underlying trends in the human's dataset. Theoretically, we prove that Counter-BC can extract a simple and similar-to-demonstration policy from imperfect data, multiple users, and teachers of varying skill levels. Empirically, we compare Counter-BC to state-of-the-art alternatives in settings with noisy demonstrations, standardized datasets, and real human teachers. Overall, we find that trying to extrapolate what the human teacher meant by considering nearby actions can lead to more proficient learning from humans. See videos of our work here: https://youtu.be/XaeOZWhTt68
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
