ManipDreamer：利用动作树与视觉引导增强机器人操作世界模型
ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance
AI总结：
ManipDreamer通过动作树建模指令关系并融合深度与语义视觉引导，显著提升机器人操作世界模型的指令跟随能力和视频质量。
摘要：
尽管近期在机器人操作视频合成方面的进展显示出潜力，但在确保有效指令跟随和实现高视觉质量方面仍存在重大挑战。近期方法如RoboDreamer利用语言分解将指令划分为独立的低层原语，并基于这些原语对世界模型进行条件化，以实现组合式指令跟随。然而，这些独立原语并未考虑它们之间存在的关联。此外，近期方法忽视了有价值的视觉引导，包括深度引导和语义引导，而这两者对于提升视觉质量至关重要。本文提出了ManipDreamer，一种基于动作树和视觉引导的先进世界模型。为了更好地学习指令原语之间的关系，我们将指令表示为动作树，并为树节点分配嵌入，每条指令可通过遍历动作树获取其嵌入。这些指令嵌入可用于引导世界模型。为了增强视觉质量，我们通过引入一个与世界模型兼容的视觉引导适配器，结合了深度引导和语义引导。该视觉适配器增强了视频生成的时间一致性和物理一致性。基于动作树和视觉引导，ManipDreamer显著提升了指令跟随能力和视觉质量。在机器人操作基准上的综合评估表明，与近期的RoboDreamer模型相比，ManipDreamer在已见和未见任务中的视频质量指标均取得了大幅提升，在未见任务中PSNR从19.55提升至21.05，SSIM从0.7474提升至0.7982，Flow Error从3.506降低至3.201。此外，我们的方法在6个RLbench任务中将机器人操作任务的成功率平均提高了2.5%。
英文摘要：
While recent advancements in robotic manipulation video synthesis have shown promise, significant challenges persist in ensuring effective instruction-following and achieving high visual quality. Recent methods, like RoboDreamer, utilize linguistic decomposition to divide instructions into separate lower-level primitives, conditioning the world model on these primitives to achieve compositional instruction-following. However, these separate primitives do not consider the relationships that exist between them. Furthermore, recent methods neglect valuable visual guidance, including depth and semantic guidance, both crucial for enhancing visual quality. This paper introduces ManipDreamer, an advanced world model based on the action tree and visual guidance. To better learn the relationships between instruction primitives, we represent the instruction as the action tree and assign embeddings to tree nodes, each instruction can acquire its embeddings by navigating through the action tree. The instruction embeddings can be used to guide the world model. To enhance visual quality, we combine depth and semantic guidance by introducing a visual guidance adapter compatible with the world model. This visual adapter enhances both the temporal and physical consistency of video generation. Based on the action tree and visual guidance, ManipDreamer significantly boosts the instruction-following ability and visual quality. Comprehensive evaluations on robotic manipulation benchmarks reveal that ManipDreamer achieves large improvements in video quality metrics in both seen and unseen tasks, with PSNR improved from 19.55 to 21.05, SSIM improved from 0.7474 to 0.7982 and reduced Flow Error from 3.506 to 3.201 in unseen tasks, compared to the recent RoboDreamer model. Additionally, our method increases the success rate of robotic manipulation tasks by 2.5% in 6 RLbench tasks on average.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
