Sketch3DVE：基于草图的3D感知场景视频编辑
Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
AI总结：
Sketch3DVE提出一种基于草图的3D感知视频编辑方法，通过点云编辑、深度图几何对齐、3D感知掩码传播和视频扩散模型，解决大视点变化下3D场景视频的结构编辑与一致性保持问题。
摘要：
近年来，视频编辑方法在风格迁移或外观修改方面取得了引人注目的成果。然而，编辑视频中3D场景的结构内容仍然具有挑战性，尤其是在处理显著视点变化（如大幅相机旋转或缩放）时。关键挑战包括生成与原始视频保持一致的新视图内容、保留未编辑区域，以及将稀疏的2D输入转换为逼真的3D视频输出。为了解决这些问题，我们提出了Sketch3DVE，一种基于草图的3D感知视频编辑方法，能够对具有显著视点变化的视频进行精细的局部操控。为解决稀疏输入带来的挑战，我们采用图像编辑方法为第一帧生成编辑结果，然后将其传播到视频的其余帧。我们利用草图作为交互工具以实现精确的几何控制，同时也支持其他基于掩码的图像编辑方法。为处理视点变化，我们对视频中的3D信息进行了详细分析和操控。具体而言，我们利用稠密立体方法估计输入视频的点云和相机参数。随后，我们提出一种点云编辑方法，使用深度图来表示新编辑组件的3D几何结构，并将其与原始3D场景有效对齐。为将新编辑内容与原始视频无缝融合，同时保留未编辑区域的特征，我们引入了一种3D感知掩码传播策略，并采用视频扩散模型生成逼真的编辑视频。大量实验证明了Sketch3DVE在视频编辑中的优越性。主页与代码：http://http://geometrylearning.com/Sketch3DVE/
英文摘要：
Recent video editing methods achieve attractive results in style transfer or appearance modification. However, editing the structural content of 3D scenes in videos remains challenging, particularly when dealing with significant viewpoint changes, such as large camera rotations or zooms. Key challenges include generating novel view content that remains consistent with the original video, preserving unedited regions, and translating sparse 2D inputs into realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a sketch-based 3D-aware video editing method to enable detailed local manipulation of videos with significant viewpoint changes. To solve the challenge posed by sparse inputs, we employ image editing methods to generate edited results for the first frame, which are then propagated to the remaining frames of the video. We utilize sketching as an interaction tool for precise geometry control, while other mask-based image editing methods are also supported. To handle viewpoint changes, we perform a detailed analysis and manipulation of the 3D information in the video. Specifically, we utilize a dense stereo method to estimate a point cloud and the camera parameters of the input video. We then propose a point cloud editing approach that uses depth maps to represent the 3D geometry of newly edited components, aligning them effectively with the original 3D scene. To seamlessly merge the newly edited content with the original video while preserving the features of unedited regions, we introduce a 3D-aware mask propagation strategy and employ a video diffusion model to produce realistic edited videos. Extensive experiments demonstrate the superiority of Sketch3DVE in video editing. Homepage and code: http://http://geometrylearning.com/Sketch3DVE/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
