虚拟存在：利用多视角表演捕捉定制可控摄像机的视频扩散模型
Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures
AI总结：
该框架通过4DGS数据管道定制视频扩散模型，实现多视角角色一致性和3D摄像机控制，支持多主体生成，提升虚拟制作中的视频质量与可控性。
摘要：
我们提出一个框架，通过新颖的定制数据管道，在视频扩散模型中实现多视角角色一致性和3D摄像机控制。我们使用4D高斯泼溅（4DGS）重新渲染的录制体积捕捉表演，并结合视频重光照模型获得的光照变化，来训练角色一致性组件。我们在此数据上微调最先进的开源视频扩散模型，以提供强大的多视角身份保持、精确的摄像机控制和光照适应性。我们的框架还支持虚拟制作的核心能力，包括使用两种方法进行多主体生成：联合训练和噪声混合，后者在推理时能够高效组合独立定制的模型；它还实现了场景和真实生活视频的定制，以及在定制过程中对运动空间布局的控制。大量实验表明，视频质量、个性化准确性、摄像机控制和光照适应性均有提升，推进了视频生成在虚拟制作中的集成。我们的项目页面可在以下网址访问：https://eyeline-labs.github.io/Virtually-Being。
英文摘要：
We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train the character consistency component with recorded volumetric capture performances re-rendered with diverse camera trajectories via 4D Gaussian Splatting (4DGS), lighting variability obtained with a video relighting model. We fine-tune state-of-the-art open-source video diffusion models on this data to provide strong multi-view identity preservation, precise camera control, and lighting adaptability. Our framework also supports core capabilities for virtual production, including multi-subject generation using two approaches: joint training and noise blending, the latter enabling efficient composition of independently customized models at inference time; it also achieves scene and real-life video customization as well as control over motion and spatial layout during customization. Extensive experiments show improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability, advancing the integration of video generation into virtual production. Our project page is available at: https://eyeline-labs.github.io/Virtually-Being.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
