arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02591cs.SDcs.AI
收藏

VividVoice: 一个统一的框架,用于场景感知的视觉驱动语音合成

VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis

Shenzhen International Graduate School, Tsinghua University, China(清华大学深圳国际研究生院);Ant Group, China(蚂蚁集团)

Chengyuan Ma, Jiawei Jin, Ruijie Xiong, Chunxiang Jin, Canxiang Yan, Wenming Yang

AI总结:

VividVoice通过构建大规模多模态数据集和设计核心对齐模块,提升了场景感知的视觉驱动语音合成的音频保真度和多模态一致性。

摘要:

我们引入并定义了一个新的任务——场景感知的视觉驱动语音合成,旨在解决现有语音生成模型在创建与现实物理世界一致的沉浸式听觉体验方面的局限性。为了解决数据稀缺性和模态解耦这两个核心挑战,我们提出了VividVoice,一个统一的生成框架。首先,我们构建了一个大规模、高质量的混合多模态数据集,Vivid-210K,通过创新的程序化管道,首次建立了视觉场景、说话人身份和音频之间的强相关性。第二,我们设计了一个核心对齐模块,D-MSVA,该模块利用解耦的记忆银行架构和跨模态混合监督策略,实现了从视觉场景到音色和环境声音特征的细粒度对齐。主客观实验结果提供了强有力的证据,表明VividVoice在音频保真度、内容清晰度和多模态一致性方面显著优于现有基线模型。我们的演示可在https://chengyuann.github.io/VividVoice/上获取。

英文摘要:

We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models in creating immersive auditory experiences that align with the real physical world. To tackle the two core challenges of data scarcity and modality decoupling, we propose VividVoice, a unified generative framework. First, we constructed a large-scale, high-quality hybrid multimodal dataset, Vivid-210K, which, through an innovative programmatic pipeline, establishes a strong correlation between visual scenes, speaker identity, and audio for the first time. Second, we designed a core alignment module, D-MSVA, which leverages a decoupled memory bank architecture and a cross-modal hybrid supervision strategy to achieve fine-grained alignment from visual scenes to timbre and environmental acoustic features. Both subjective and objective experimental results provide strong evidence that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency. Our demo is available at https://chengyuann.github.io/VividVoice/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑