DRAW2ACT：将深度编码轨迹转化为机器人演示视频
DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos
AI总结：
DRAW2ACT通过深度感知的轨迹条件视频生成框架，生成可控且一致的机器人演示视频，提升机器人操作的成功率。
摘要：
视频扩散模型为具身AI提供了强大的现实世界模拟器，但其在机器人操作的可控性方面仍存在局限。最近关于轨迹条件视频生成的研究解决了这一差距，但通常依赖于二维轨迹或单模态条件，限制了其生成可控且一致的机器人演示的能力。我们提出了DRAW2ACT，一种深度感知的轨迹条件视频生成框架，该框架从输入轨迹中提取多种正交表示，捕捉深度、语义、形状和运动，并将它们注入扩散模型。此外，我们提出联合生成空间对齐的RGB和深度视频，利用跨模态注意力机制和深度监督来增强时空一致性。最后，我们引入了一个基于生成RGB和深度序列的多模态策略模型，以回归机器人的关节角度。在Bridge V2、伯克利Autolab和模拟基准测试中，DRAW2ACT在视觉保真度和一致性方面优于现有基线，同时产生了更高的操作成功率。
英文摘要：
Video diffusion models provide powerful real-world simulators for embodied AI but remain limited in controllability for robotic manipulation. Recent works on trajectory-conditioned video generation address this gap but often rely on 2D trajectories or single modality conditioning, which restricts their ability to produce controllable and consistent robotic demonstrations. We present DRAW2ACT, a depth-aware trajectory-conditioned video generation framework that extracts multiple orthogonal representations from the input trajectory, capturing depth, semantics, shape and motion, and injects them into the diffusion model. Moreover, we propose to jointly generate spatially aligned RGB and depth videos, leveraging cross-modality attention mechanisms and depth supervision to enhance the spatio-temporal consistency. Finally, we introduce a multimodal policy model conditioned on the generated RGB and depth sequences to regress the robot's joint angles. Experiments on Bridge V2, Berkeley Autolab, and simulation benchmarks show that DRAW2ACT achieves superior visual fidelity and consistency while yielding higher manipulation success rates compared to existing baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
