UniTalking: 一种统一的音频-视频框架用于说话肖像生成
UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation
AI总结：
UniTalking提出了一种统一的端到端扩散框架，用于生成高质量的语音和唇同步视频，通过多模态Transformer块和个性化语音克隆技术，提升了视觉保真度和训练效率。
摘要：
尽管最先进的音频-视频生成模型如Veo3和Sora2展示了非凡的能力，但其闭源性质使得其架构和训练范式难以获取。为弥合在可访问性和性能之间的差距，我们引入了UniTalking，一种统一的端到端扩散框架，用于生成高质量的语音和唇同步视频。在核心部分，我们的框架采用多模态Transformer块，通过共享的自注意力机制显式建模音频和视频潜在标记之间的细粒度时间对应关系。通过利用预训练视频生成模型的强大先验知识，我们的框架在保证最先进的视觉保真度的同时，实现了高效的训练。此外，UniTalking集成了个性化语音克隆能力，允许从简短的音频参考中生成目标风格的语音。定性和定量结果表明，我们的方法生成了高度逼真的说话肖像，在唇同步准确性、音频自然度和整体感知质量方面均优于现有开源方法。
英文摘要：
While state-of-the-art audio-video generation models like Veo3 and Sora2 demonstrate remarkable capabilities, their closed-source nature makes their architectures and training paradigms inaccessible. To bridge this gap in accessibility and performance, we introduce UniTalking, a unified, end-to-end diffusion framework for generating high-fidelity speech and lip-synchronized video. At its core, our framework employs Multi-Modal Transformer Blocks to explicitly model the fine-grained temporal correspondence between audio and video latent tokens via a shared self-attention mechanism. By leveraging powerful priors from a pre-trained video generation model, our framework ensures state-of-the-art visual fidelity while enabling efficient training. Furthermore, UniTalking incorporates a personalized voice cloning capability, allowing the generation of speech in a target style from a brief audio reference. Qualitative and quantitative results demonstrate that our method produces highly realistic talking portraits, achieving superior performance over existing open-source approaches in lip-sync accuracy, audio naturalness, and overall perceptual quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
