通过流反转引导改进机器人通用策略
Improving Robotic Generalist Policies via Flow Reversal Steering
AI总结:
提出流反转引导(FRS)方法,通过逆向流策略找到次优动作的潜在噪声并映射到通用策略的动作模式,提升零样本控制、行为克隆和强化学习效果。
摘要:
通用策略可以从多样化的机器人数据集中学习广泛的技能。为了解决或改进具有挑战性的新任务,我们需要一种方法从策略丰富的行为先验中推断并调用适当的动作,特别是当直接命令策略失败时。我们专注于流匹配通用策略,并提出流反转引导(FRS):一种方法,它采用次优但“合理”的动作,通过逆向流策略传递它们以找到其潜在噪声,并将它们映射到附近的通用策略动作模式。我们在多个模拟和真实世界的操作设置中评估了FRS。首先,FRS可以将来自人类或视觉语言模型的粗略语义引导转化为相应的良好机器人动作,从而改进零样本控制。这些收益可以通过行为克隆进行蒸馏,通过训练一个辅助策略输出噪声,通用策略将其映射到良好动作——在不到一分钟的训练中显示出高达95%的绝对任务成功率提升。最后,FRS通过用语义知识引导强化学习实现策略改进,在标准强化学习无法改进的多个任务上取得了改进。
英文摘要:
Generalist policies can learn a wide range of skills from diverse robot datasets. In order to solve or improve on challenging new tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior, especially when directly commanding the policy fails. We focus on flow matching generalists and propose Flow Reversal Steering (FRS): a method that takes suboptimal but ``reasonable'' actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes. We evaluate FRS across many simulated and real-world manipulation settings. First, FRS can turn coarse semantic guidance from humans or vision-language models (VLMs) into corresponding good robot actions, improving zero-shot control. These gains can be distilled with behavioral cloning by training an auxiliary policy to output noises that the generalist maps to good actions -- showing up to 95% absolute task success rate boosts in under a minute of training. Finally, FRS enables policy improvement by bootstrapping reinforcement learning with semantic knowledge, improving on several tasks that standard RL fails to improve on.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。