DexSIM: 具有统一因果视频扩散的实时灵巧仿真
DexSIM: Real-time Dexterous Simulation with Unified Causal Video Diffusion
AI总结：
提出DexSIM框架，通过两阶段训练（双向视频扩散和自回归滚动训练）实现实时、长时一致的灵巧操作仿真，在像素相似度、运动保真度和手部投影精度上超越基线。
摘要：
视频扩散模型的最新进展已实现对物理世界的大规模仿真，但手部物体交互的仿真研究较少。我们提出DexSIM，一个用于实时灵巧操作仿真的灵巧仿真框架。以往利用视频扩散和3D重建的工作侧重于导航，而灵巧操作虽在创建交互式仿真体验和为机器人生成合成数据方面有广泛应用，但进展有限。现有方法缺乏实时交互性、长期空间一致性和记忆。我们为DexSIM提出两阶段训练框架。首先，通过在手部动作轨迹和视频的统一特征空间中进行联合嵌入，训练一个双向视频扩散模型。我们利用高斯热图手部编码实现更准确的手部表示。然后，我们进行基于滚动的自回归训练，将更新的空间缓存作为注意力汇点用于空间记忆，从而提高了长期一致性和3D感知灵巧操作仿真。DexSIM在像素和语义相似度、运动保真度和手部投影精度上优于基线。它还支持手部运动迁移等新应用，并以15.24 FPS的帧率实现实时交互。
英文摘要：
Recent progress of video diffusion models have enabled extensive simulation of the physical world. While simulation with hand object interaction has been less explored. We propose DexSIM, a dexterous simulation framework for simulating dexterous manipulation in real-time. While previous works utilizing video diffusion and 3D reconstruction focus on navigation, dexterous manipulation has been limited while it has extensive applications for creating interactive experiences with the simulated world and for generating synthetic data for robotics. Existing methods lack real-time interactivity and long-term spatial consistency and memory. We propose a 2-stage training framework for DexSIM. First we train a bi-directional video diffusion model by jointly embedding the hand action trajectory and video in a unified feature space. We utilize gaussian heatmap hand encoding for more accurate hand representation. Then we conduct a roll-out based autoregressive training with updated spatial cache as attention sink for spatial memory, which improves long-term consistency and 3D aware dexterous manipulation simulation. DexSIM outperforms the baseline on pixel and semantic similarity, motion fidelity, and hand projection accuracy. It also allows new applications such as hand motion transfer and runs at 15.24 FPS real-time interactivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
