EditYourself: 基于音频驱动的谈话头视频生成与操控的扩散变换器
EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers
AI总结：
EditYourself通过音频驱动的视频到视频编辑框架，实现基于脚本的谈话头视频修改，包括内容的添加、删除和重新定时，同时保持唇同步和时间一致性。
摘要：
当前的生成视频模型在从文本和图像提示生成新内容方面表现出色，但在编辑现有预录制视频方面存在关键缺口，其中对语音脚本的微小修改需要保持运动、时间一致性、说话者身份和准确的唇同步。我们介绍了EditYourself，一种基于DiT的框架，用于音频驱动的视频到视频（V2V）编辑，能够基于脚本对谈话头视频进行修改，包括无缝添加、删除和重新定时视觉语音内容。在通用视频扩散模型的基础上，EditYourself通过音频条件和区域感知、以编辑为中心的训练扩展增强了其V2V能力。这使得可以通过时空修复实现精确的唇同步和现有表演的时间一致性重构，包括在新增部分合成逼真的人体运动，同时在长时间内保持视觉保真度和身份一致性。这项工作是生成视频模型作为专业视频后期制作实用工具的基石性步骤。
英文摘要：
Current generative video models excel at producing novel content from text and image prompts, but leave a critical gap in editing existing pre-recorded videos, where minor alterations to the spoken script require preserving motion, temporal coherence, speaker identity, and accurate lip synchronization. We introduce EditYourself, a DiT-based framework for audio-driven video-to-video (V2V) editing that enables transcript-based modification of talking head videos, including the seamless addition, removal, and retiming of visually spoken content. Building on a general-purpose video diffusion model, EditYourself augments its V2V capabilities with audio conditioning and region-aware, edit-focused training extensions. This enables precise lip synchronization and temporally coherent restructuring of existing performances via spatiotemporal inpainting, including the synthesis of realistic human motion in newly added segments, while maintaining visual fidelity and identity consistency over long durations. This work represents a foundational step toward generative video models as practical tools for professional video post-production.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
