OmniTalker：具备多模态风格模仿的一次性实时文本驱动说话音视频生成
OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking
AI总结：
OmniTalker提出双分支扩散Transformer框架，首次实现一次性实时文本驱动的说话音视频联合生成，可同时模仿语音与面部风格并保持音视频同步。
摘要：
尽管音频驱动的说话头部生成已取得显著进展，但文本驱动方法仍未被充分探索。本文提出了OmniTalker，一个统一框架，能够根据输入文本联合生成同步的说话音视频内容，同时模仿目标身份的语言和面部运动风格，包括语音特征、头部运动和面部动态。该框架采用双分支扩散Transformer（DiT）架构，一个分支用于音频生成，另一个分支用于视频合成。在浅层中引入跨模态融合模块以整合两种模态之间的信息；在深层中，各模态独立处理，生成的音频由声码器解码，视频则使用基于GAN的高质量视觉渲染器进行渲染。借助DiT的上下文学习能力，并通过掩码填充策略，模型能够同时捕捉音频和视觉风格，而无需显式的风格提取模块。得益于DiT主干网络的高效性以及优化的视觉渲染器，OmniTalker实现了25 FPS的实时推理。据我们所知，OmniTalker是第一个能够实时联合建模语音和面部风格的一次性框架。大量实验表明，该方法在生成质量上优于现有方法，尤其在保持风格一致性和确保精确的音视频同步方面表现突出，同时保持了高效的推理。
英文摘要：
Although significant progress has been made in audio-driven talking head generation, text-driven methods remain underexplored. In this work, we present OmniTalker, a unified framework that jointly generates synchronized talking audio-video content from input text while emulating the speaking and facial movement styles of the target identity, including speech characteristics, head motion, and facial dynamics. Our framework adopts a dual-branch diffusion transformer (DiT) architecture, with one branch dedicated to audio generation and the other to video synthesis. At the shallow layers, cross-modal fusion modules are introduced to integrate information between the two modalities. In deeper layers, each modality is processed independently, with the generated audio decoded by a vocoder and the video rendered using a GAN-based high-quality visual renderer. Leveraging the in-context learning capability of DiT through a masked-infilling strategy, our model can simultaneously capture both audio and visual styles without requiring explicit style extraction modules. Thanks to the efficiency of the DiT backbone and the optimized visual renderer, OmniTalker achieves real-time inference at 25 FPS. To the best of our knowledge, OmniTalker is the first one-shot framework capable of jointly modeling speech and facial styles in real time. Extensive experiments demonstrate its superiority over existing methods in terms of generation quality, particularly in preserving style consistency and ensuring precise audio-video synchronization, all while maintaining efficient inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
