面向视图自适应的人体视频生成的3D感知隐式运动控制
3D-Aware Implicit Motion Control for View-Adaptive Human Video Generation
AI总结：
3DiMo通过隐式运动表示和视图丰富监督，提升视频生成中的人体运动保真度和视觉质量。
摘要：
现有的人体运动控制方法在视频生成中通常依赖于2D姿态或显式3D参数模型（如SMPL）作为控制信号。然而，2D姿态将运动严格绑定到驱动视角，阻碍了新视角合成。尽管显式3D模型在结构上信息丰富，但其固有的不准确性（如深度模糊和动态不准确）在作为强约束时会覆盖大规模视频生成器强大的内在3D意识。在本文中，我们从3D感知的角度重新审视运动控制，倡导一种隐式、视图无关的运动表示，这种表示自然与生成器的空间先验对齐，而不是依赖外部重建的约束。我们引入了3DiMo，它联合训练了一个运动编码器和预训练的视频生成器，将驱动帧压缩成紧凑的、视图无关的运动标记，并通过交叉注意力注入语义。为了促进3D感知，我们使用视图丰富的监督（即单视角、多视角和移动相机视频），迫使运动在不同视角下保持一致。此外，我们使用辅助几何监督，仅在早期初始化中利用SMPL，并将其衰减到零，使模型能够从数据和生成器的先验中过渡到学习真实的3D空间运动理解。实验表明，3DiMo能够忠实再现驱动运动，具有灵活的文本驱动相机控制，显著优于现有方法在运动保真度和视觉质量方面。
英文摘要：
Existing methods for human motion control in video generation typically rely on either 2D poses or explicit 3D parametric models (e.g., SMPL) as control signals. However, 2D poses rigidly bind motion to the driving viewpoint, precluding novel-view synthesis. Explicit 3D models, though structurally informative, suffer from inherent inaccuracies (e.g., depth ambiguity and inaccurate dynamics) which, when used as a strong constraint, override the powerful intrinsic 3D awareness of large-scale video generators. In this work, we revisit motion control from a 3D-aware perspective, advocating for an implicit, view-agnostic motion representation that naturally aligns with the generator's spatial priors rather than depending on externally reconstructed constraints. We introduce 3DiMo, which jointly trains a motion encoder with a pretrained video generator to distill driving frames into compact, view-agnostic motion tokens, injected semantically via cross-attention. To foster 3D awareness, we train with view-rich supervision (i.e., single-view, multi-view, and moving-camera videos), forcing motion consistency across diverse viewpoints. Additionally, we use auxiliary geometric supervision that leverages SMPL only for early initialization and is annealed to zero, enabling the model to transition from external 3D guidance to learning genuine 3D spatial motion understanding from the data and the generator's priors. Experiments confirm that 3DiMo faithfully reproduces driving motions with flexible, text-driven camera control, significantly surpassing existing methods in both motion fidelity and visual quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
