面向音视频配音的视频编辑
Video Editing for Audio-Visual Dubbing
AI总结：
本研究提出EdiDub框架，将视觉配音重新表述为内容感知编辑任务，通过专门的调节方案保留原始视频上下文，在多个基准测试中显著提升了身份保持和唇部同步效果，优于传统生成与修复方法。
摘要：
视觉配音，即面部动作与新语音的同步，对于使内容跨语言可及并实现更广泛的全球传播至关重要。然而，当前方法面临重大局限。现有方法通常生成说话人脸，阻碍了其与原始场景的无缝融合，或采用修复技术，丢弃了部分遮挡和光照变化等关键视觉信息。本研究引入了EdiDub，一个将视觉配音重新表述为内容感知编辑任务的新颖框架。EdiDub通过利用专门的调节方案来保留原始视频上下文，以确保忠实准确的修改，而非简单复制。在多个基准测试（包括一个具有挑战性的遮挡唇部数据集）上，EdiDub显著提升了身份保持和同步性。人类评估进一步证实了其优越性，与领先方法相比，获得了更高的同步性和视觉自然度评分。这些结果表明，我们的内容感知编辑方法优于传统的生成或修复，特别是在保持复杂视觉元素的同时确保准确的唇部同步。
英文摘要：
Visual dubbing, the synchronization of facial movements with new speech, is crucial for making content accessible across different languages, enabling broader global reach. However, current methods face significant limitations. Existing approaches often generate talking faces, hindering seamless integration into original scenes, or employ inpainting techniques that discard vital visual information like partial occlusions and lighting variations. This work introduces EdiDub, a novel framework that reformulates visual dubbing as a content-aware editing task. EdiDub preserves the original video context by utilizing a specialized conditioning scheme to ensure faithful and accurate modifications rather than mere copying. On multiple benchmarks, including a challenging occluded-lip dataset, EdiDub significantly improves identity preservation and synchronization. Human evaluations further confirm its superiority, achieving higher synchronization and visual naturalness scores compared to the leading methods. These results demonstrate that our content-aware editing approach outperforms traditional generation or inpainting, particularly in maintaining complex visual elements while ensuring accurate lip synchronization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
