ManipDreamer3D：利用感知占据的3D轨迹合成可信的机器人操作视频
ManipDreamer3D : Synthesizing Plausible Robotic Manipulation Video with Occupancy-aware 3D Trajectory
AI总结：
针对机器人操作领域数据稀缺、现有视频生成方法依赖2D轨迹存在3D空间歧义的问题，提出ManipDreamer3D框架，结合3D占据感知轨迹规划与轨迹到视频扩散模型，生成高质量可信机器人操作视频，减少人工干预。
摘要：
数据稀缺始终是机器人操作领域的一大核心挑战。尽管扩散模型为生成机器人操作视频提供了颇具潜力的解决方案，但现有方法大多依赖2D轨迹，这类方法天然存在3D空间歧义性问题。本研究提出名为ManipDreamer3D的新型框架，可基于输入图像与文本指令生成具备3D感知能力的可信机器人操作视频。该方法将3D轨迹规划与从第三人称视角重建的3D占据图相结合，同时搭配全新的轨迹到视频扩散模型。具体而言，ManipDreamer3D首先从输入图像重建3D占据表示，随后计算优化后的3D末端执行器轨迹，在规避碰撞的同时最小化路径长度。接下来，我们采用隐式编辑技术，基于初始图像隐空间与优化后的3D轨迹生成视频序列，该过程以我们专门训练的轨迹到视频扩散模型为条件，生成机器人抓取放置视频。该方法可生成具备自主规划的可信3D轨迹的机器人视频，大幅降低人工干预需求。实验结果表明，其视觉质量优于现有方法。
英文摘要：
Data scarcity continues to be a major challenge in the field of robotic manipulation. Although diffusion models provide a promising solution for generating robotic manipulation videos, existing methods largely depend on 2D trajectories, which inherently face issues with 3D spatial ambiguity. In this work, we present a novel framework named ManipDreamer3D for generating plausible 3D-aware robotic manipulation videos from the input image and the text instruction. Our method combines 3D trajectory planning with a reconstructed 3D occupancy map created from a third-person perspective, along with a novel trajectory-to-video diffusion model. Specifically, ManipDreamer3D first reconstructs the 3D occupancy representation from the input image and then computes an optimized 3D end-effector trajectory, minimizing path length while avoiding collisions. Next, we employ a latent editing technique to create video sequences from the initial image latent and the optimized 3D trajectory. This process conditions our specially trained trajectory-to-video diffusion model to produce robotic pick-and-place videos. Our method generates robotic videos with autonomously planned plausible 3D trajectories, significantly reducing human intervention requirements. Experimental results demonstrate superior visual quality compared to existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
