Real2SAM2Real: 生成式3D缓存作为视频扩散的互补上下文
Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion
AI总结：
提出Real2SAM2Real框架，通过3D提升模型提取可编辑的3D缓存作为几何支架，结合软空间对齐注入和微调策略，实现视频扩散模型对相机轨迹和多实体运动的精确解耦控制。
摘要：
虽然视频扩散模型（VDM）在合成高保真视频方面表现出色，但实现精确的相机和场景控制仍然具有挑战性。现有方法主要依赖隐式扩散先验来生成未观察区域，在高动态运动或复杂遮挡期间不可避免地导致结构崩溃。为了解决这一挑战，我们提出了Real2SAM2Real框架，该框架利用3D提升模型（例如SAM3D）提取显式可编辑的3D缓存，作为VDM的稳健几何支架。通过捕获前景实体的整个3D体积而不仅仅是其可见外壳，该缓存将整体空间先验注入VDM，为复杂场景动态提供可靠的3D感知指导。为了有效利用这种3D指导同时保留预训练先验，我们设计了一种软空间对齐注入机制以及一种针对VDM量身定制的微创微调策略。此外，我们采用掩码法线图作为跨模态桥梁，构建了无3D数据的数据整理和扰动流程。大量实验表明，Real2SAM2Real能够对相机轨迹和多实体运动实现精确、解耦的控制。通过利用生成式3D缓存的互补上下文，我们的框架克服了因过度依赖扩散先验而导致的典型崩溃，在大的相机位移和严重遮挡下保持了卓越的时空一致性。关键的是，通过将几何与外观解耦，我们为VDM定制的3D缓存消除了由结构空洞和错误立面引起的视角歧义，以及反射和折射引起的误导性线索。项目网站见https://jiayi-wu-leo.github.io/real2sam2real。
英文摘要：
Video Diffusion Models (VDMs) synthesize high-quality videos, but precise camera and motion control remains challenging when large viewpoint shifts, changing occlusions, or complex dynamics reveal unseen regions. Diffusion priors alone can yield structural inconsistencies, while warped guidance can contain incomplete geometry and misleading view-dependent appearance cues. Real2SAM2Real complements VDM priors with a coarse 3D foreground cache of topologically complete meshes for intuitive editing and animation. Generative 3D lifting estimates these object and human meshes, including unobserved surfaces, from a single reference image. Normal maps rendered under specified camera trajectories and foreground motions guide video synthesis in the VDM, while the reference image provides appearance conditioning. Soft spatial-aligned injection and minimally invasive fine-tuning incorporate this guidance while largely preserving pretrained synthesis priors. Decoupled conditioning enables independent appearance and geometry editing. We train on around 2,000 monocular clips using instance-masked normal sequences with instance-level spatial perturbations to reduce sensitivity to differences between video-estimated training normals and normals rendered from coarse 3D caches at inference. Training requires neither explicit 3D assets nor paired multi-trajectory recordings. Real2SAM2Real supports decoupled camera and multi-instance motion control, including articulated humans. Across these challenging settings, experiments show strong spatiotemporal consistency and better overall control accuracy and video fidelity than evaluated state-of-the-art methods. Preliminary demonstrations suggest potential robotics applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
