发表机构
Monash University; University of Chinese Academy of Sciences; Tsinghua University（莫纳什大学; 中国科学院大学; 清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出基于扩散模型的统一架构CineDub，无需人脸裁剪或说话人分割即可实现精准多说话人对话配音，还引入两项训练策略并发布两个野外基准，在相关任务中取得最优性能。
AI 中文摘要
野外环境下的自动视频配音仍受两个相互制约的核心问题限制：分层式方法依赖脆弱的多阶段预处理流水线，严重限制了数据扩展性与实际部署能力；而针对未裁剪视频的整体式方法，在多说话人场景中存在时间对齐精度不足、说话人-语句歧义等问题。为克服这些局限，我们提出CineDub，这是一种基于扩散模型的统一架构，可直接从未裁剪视频实现精准的多说话人对话配音，无需人脸裁剪或说话人 diarization（说话人分割）。我们方法的核心是隐式耦合整体条件（ICHC）范式：整体视觉表征与语义打包的转录格式被独立编码，却通过跨模态训练实现隐式耦合，以解决说话人歧义并支持精准的多说话人多轮对话配音。基于整体视觉特征捕捉的统一时间线索，我们进一步将CineDub扩展至语音与音频的联合生成。我们引入环境-语言课程学习（ALC）以缓解子任务性能退化，还采用解耦文本分支控制机制，解决同时生成过程中的跨提示干扰问题。我们还发布了两个野外基准：用于多说话人对话配音的CineDub-Multi，以及用于视频转语音与音频（V2SA）生成的CineDub-SA，以支持真实场景下的评估。实验表明，CineDub在已有的单说话人配音和视频转音频基准上达到了SOTA（当前最优）结果，且在多说话人对话配音与声学连贯的联合生成任务中表现尤为出色。
英文摘要
Automatic video dubbing in the wild remains fundamentally limited by two competing constraints: hierarchical methods depend on brittle, multi-stage preprocessing pipelines that severely restrict data scalability and practical deployment, while holistic approaches operating on uncropped video suffer from weak temporal alignment and speaker-utterance ambiguity in multi-speaker settings. To overcome these limitations, we propose CineDub, a unified diffusion-based model that achieves precise multi-speaker dialogue dubbing directly from uncropped videos, without face cropping or speaker diarization. Central to our approach is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm, where holistic visual representations and a semantic-bundled transcription format are encoded independently, yet implicitly coupled through cross-modal training to resolve speaker ambiguity and enable precise multi-speaker multi-turn dialogue dubbing. Building on the unified temporal cues captured by holistic visual features, we further extend CineDub to joint speech and audio generation. We introduce an Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation, and a decoupled textual branch control mechanism to resolve cross-prompt interference during simultaneous generation. We also release two in-the-wild benchmarks, CineDub-Multi for multi-speaker dialogue dubbing and CineDub-SA for video-to-speech-and-audio (V2SA) generation, to enable evaluation under realistic conditions. Experiments show that CineDub achieves state-of-the-art results on established single-speaker dubbing and video-to-audio benchmarks while excelling in multi-speaker dialogue dubbing and acoustically coherent joint generation.
CommentsAccepted to ACM MM 2026
