使用角色-环境协调视频生成模型的电影级合成
Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models
AI总结：
提出端到端视频扩散框架，通过三掩码引导和RGB-D联合去噪建模角色与环境的双向物理与光照交互，实现高质量动态视频合成。
摘要：
电影级合成旨在将绿幕角色融入新环境，同时保持物理和光度真实性。先前的方法通常未能捕捉角色与其周围环境之间的复杂双向交互，我们将其表征为角色到环境（C2E）的物理交互和环境到角色（E2C）的光照协调。为了解决这个问题，我们提出了一个端到端的视频扩散框架，联合建模C2E和E2C交互，特别处理交互道具的挑战。我们的方法引入了一种三掩码引导架构，结合RGB-D联合去噪，以确保角色、道具和环境之间的物理一致交互。我们进一步开发了一种高效的先验驱动数据整理流程，无需昂贵的渲染即可构建高质量的重光照对。最后，参考条件机制实现了可控的环境合成和精确的道具替换。大量实验表明，我们的框架在电影级动态视频合成方面显著优于现有方法。
英文摘要：
Cinematic compositing aims to integrate green-screen characters into novel environments while maintaining physical and photometric realism. Previous methods often fail to capture the complex bidirectional interactions between characters and their surroundings, which we characterize as Character-to-Environment (C2E) physical interaction and Environment-to-Character (E2C) lighting harmonization. To address this, we propose an end-to-end video diffusion framework that jointly models C2E and E2C interactions, specifically handling the challenges of interactive props. Our approach introduces a tri-mask-guided architecture with RGB-D joint denoising to ensure physically consistent interactions among the character, props, and environment. We further develop an efficient prior-driven data curation pipeline to construct high-quality relighting pairs without expensive rendering. Finally, a reference-conditioned mechanism enables controllable environment synthesis and precise prop replacement. Extensive experiments demonstrate that our framework significantly outperforms existing methods in cinematic-quality dynamic video compositing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
