MVHOI: 通过3D基础模型桥接多视角条件与复杂人-物体交互视频重现
MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model
AI总结:
本文提出MVHOI框架,通过3D基础模型结合多视角参考条件,生成复杂人-物体交互视频,提升了长时视频生成的性能。
摘要:
Human-Object Interaction (HOI)视频重现中真实运动仍属表达数字人类创作的前沿领域。现有方法主要处理简单图像平面运动(如平面内平移),难以处理复杂非平面操作如平面外重定向。本文提出MVHOI,一种两阶段HOI视频重现框架,通过3D基础模型(3DFM)桥接多视角参考条件与视频基础模型。3DFM首先根据隐式运动动态在新视角下生成视图一致的对象先验。可控视频生成模型随后通过整合多视角参考图像合成高保真物体纹理,通过合理检索机制确保外观一致性。通过在推理阶段使这两个阶段相互增强,我们的框架在生成包含复杂物体操作的长时HOI视频中表现出优越性能。大量实验显示,与先前方法相比,特别是在复杂3D物体操作的HOI中,有显著改进。
英文摘要:
Human-Object Interaction (HOI) video reenactment aims to transfer the interaction dynamics of a source video to a novel target object while preserving realistic hand-object coordination. Existing methods typically rely on sparse 2D motion controls and monocular references, which are insufficient for complex out-of-plane motion and large viewpoint changes. We present MVHOI, a two-stage framework combining implicit motion extraction, 3D-aware multi-view reasoning, and video generation. In the first stage, a motion extractor encodes object dynamics into implicit motion descriptors. Conditioned on these descriptors, our Motion-Driven Object Prior (MDOP) module queries a 3D foundation model over multi-view references of the target object and autoregressively predicts coarse object anchors, a sequence of images that track the object's evolving orientation and appearance under the source motion without any explicit pose estimation. In the second stage, a DiT-based video generation model uses these anchors as structural guidance and the multi-view references as appearance guidance. We further reuse cross-view attention from MDOP as a soft attention bias to reduce reference-view confusion. For long videos, a cross-iterative inference strategy refreshes subsequent object priors using refined video outputs. Experiments demonstrate consistent improvements over state-of-the-art methods in object fidelity, motion consistency, visual quality, and interaction realism.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。