OmniSync：基于扩散Transformer实现通用唇形同步
OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers
AI总结：
OmniSync提出基于扩散Transformer的无掩码唇形同步框架，结合流匹配渐进噪声初始化和动态时空无分类器引导，并构建AIGC-LipSync基准，在真实与AI生成视频中显著提升唇形同步质量。
摘要：
唇形同步任务旨在使视频中说话者的唇部动作与相应的语音音频对齐，这对于生成逼真且富有表现力的视频内容至关重要。然而，现有方法通常依赖参考帧和掩码帧修复，这限制了它们在身份一致性、姿态变化、面部遮挡和风格化内容方面的鲁棒性。此外，由于音频信号提供的条件约束弱于视觉线索，原始视频中的唇形泄漏会影响唇形同步质量。本文提出OmniSync，一个面向多样化视觉场景的通用唇形同步框架。我们的方法引入了一种无掩码训练范式，利用扩散Transformer模型进行直接帧编辑而无需显式掩码，从而支持无限时长推理，同时保持自然的面部动态并保留角色身份。在推理阶段，我们提出一种基于流匹配的渐进式噪声初始化方法，以确保姿态和身份一致性，同时允许精确的嘴部区域编辑。为解决音频条件信号较弱的问题，我们开发了一种动态时空无分类器引导（DS-CFG）机制，可在时间和空间上自适应调整引导强度。我们还建立了AIGC-LipSync Benchmark，这是首个针对多样化AI生成视频中唇形同步的评估套件。大量实验表明，OmniSync在视觉质量和唇形同步精度方面均显著优于先前方法，在真实视频和AI生成视频中均取得了更优结果。
英文摘要：
Lip synchronization is the task of aligning a speaker's lip movements in video with corresponding speech audio, and it is essential for creating realistic, expressive video content. However, existing methods often rely on reference frames and masked-frame inpainting, which limit their robustness to identity consistency, pose variations, facial occlusions, and stylized content. In addition, since audio signals provide weaker conditioning than visual cues, lip shape leakage from the original video will affect lip sync quality. In this paper, we present OmniSync, a universal lip synchronization framework for diverse visual scenarios. Our approach introduces a mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks, enabling unlimited-duration inference while maintaining natural facial dynamics and preserving character identity. During inference, we propose a flow-matching-based progressive noise initialization to ensure pose and identity consistency, while allowing precise mouth-region editing. To address the weak conditioning signal of audio, we develop a Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism that adaptively adjusts guidance strength over time and space. We also establish the AIGC-LipSync Benchmark, the first evaluation suite for lip synchronization in diverse AI-generated videos. Extensive experiments demonstrate that OmniSync significantly outperforms prior methods in both visual quality and lip sync accuracy, achieving superior results in both real-world and AI-generated videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
