Talker-T2AV：基于自回归扩散模型的联合说话音频视频生成
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
AI总结：
本文提出Talker-T2AV，通过共享骨干和模态特定解码器实现音频视频联合生成，提升跨模态一致性与生成效率。
摘要：
联合音频视频生成模型表明，统一生成比级联方法能产生更强的跨模态一致性。然而，现有模型通过普遍注意力机制在去噪过程中耦合模态，将高层语义和低层细节完全纠缠，这对说话人合成不利：虽然音频和面部运动有语义关联，但其低层实现（声学信号和视觉纹理）遵循不同的渲染过程。强制跨所有层次联合建模导致不必要的纠缠并降低效率。我们提出Talker-T2AV，一种自回归扩散框架，其中高层跨模态建模发生在共享骨干中，而低层细化使用模态特定解码器。共享自回归语言模型在统一的patch级token空间中联合推理由音频和视频组成。两个轻量级扩散Transformer头将隐藏状态解码为帧级音频和视频潜在表示。在说话人肖像基准测试中，Talker-T2AV在唇同步准确性、视频质量和音频质量上优于双分支基线，实现比级联管道更强的跨模态一致性。
英文摘要：
Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
