通过使用3D Gaussian Splatting重建视频来增强视频编辑中的时间一致性
Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splatting
AI总结：
提出Video-3DGS，一种基于3D Gaussian Splatting的两阶段视频细化器，通过MC-COLMAP生成点云并施加时间约束，显著提升了视频重构质量与训练效率，并确保了视频编辑的时间一致性。
摘要：
零样本视频扩散模型的最新进展在文本驱动的视频编辑中展现出潜力，但在实现高时间一致性方面仍存在挑战。为了解决这一问题，我们引入了Video-3DGS，这是一种基于3D Gaussian Splatting (3DGS)的视频细化器，旨在增强零样本视频编辑器的时间一致性。我们的方法利用专为编辑动态单目视频定制的两阶段3D Gaussian优化过程。在第一阶段，Video-3DGS采用了改进版的COLMAP（称为MC-COLMAP），它使用Masked和Clipped方法处理原始视频。对于每个视频片段，MC-COLMAP为动态前景物体和复杂背景生成点云。这些点云用于初始化两组3D Gaussians（Frg-3DGS和Bkg-3DGS），旨在表示前景和背景视图。然后将前景和背景视图与2D可学习参数图合并，以重建完整视图。在第二阶段，我们利用第一阶段开发的重构能力对视频扩散模型施加时间约束。为了证明Video-3DGS在两个阶段的有效性，我们在两个相关任务上进行了大量实验：视频重构和视频编辑。在DAVIS数据集上，使用3k次迭代训练的Video-3DGS显著提高了视频重构质量（+3 PSNR，+7 PSNR提升）和训练效率（分别比基于NeRF和基于3DGS的最先进方法快1.9倍、4.5倍）。此外，它通过确保58个动态单目视频的时间一致性来增强视频编辑。
英文摘要：
Recent advancements in zero-shot video diffusion models have shown promise for text-driven video editing, but challenges remain in achieving high temporal consistency. To address this, we introduce Video-3DGS, a 3D Gaussian Splatting (3DGS)-based video refiner designed to enhance temporal consistency in zero-shot video editors. Our approach utilizes a two-stage 3D Gaussian optimizing process tailored for editing dynamic monocular videos. In the first stage, Video-3DGS employs an improved version of COLMAP, referred to as MC-COLMAP, which processes original videos using a Masked and Clipped approach. For each video clip, MC-COLMAP generates the point clouds for dynamic foreground objects and complex backgrounds. These point clouds are utilized to initialize two sets of 3D Gaussians (Frg-3DGS and Bkg-3DGS) aiming to represent foreground and background views. Both foreground and background views are then merged with a 2D learnable parameter map to reconstruct full views. In the second stage, we leverage the reconstruction ability developed in the first stage to impose the temporal constraints on the video diffusion model. To demonstrate the efficacy of Video-3DGS on both stages, we conduct extensive experiments across two related tasks: Video Reconstruction and Video Editing. Video-3DGS trained with 3k iterations significantly improves video reconstruction quality (+3 PSNR, +7 PSNR increase) and training efficiency (x1.9, x4.5 times faster) over NeRF-based and 3DGS-based state-of-art methods on DAVIS dataset, respectively. Moreover, it enhances video editing by ensuring temporal consistency across 58 dynamic monocular videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
