面向沉浸式视觉文本到语音的多源空间知识理解
Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speech
AI总结：
针对VTTS仅依赖RGB模态忽略空间知识的问题，提出MS2KU-VTTS方案，通过串行交互机制动态整合RGB主导源与深度、位置和语义补充源，有效提升了沉浸式语音合成质量。
摘要：
视觉文本到语音（VTTS）旨在将环境图像作为提示，为语音内容合成混响语音。以往工作侧重于RGB模态进行全局环境建模，忽略了深度、说话者位置和环境语义等多源空间知识的潜力。为解决这些问题，我们提出一种用于沉浸式VTTS的新颖多源空间知识理解方案，称为MS2KU-VTTS。具体而言，我们首先将RGB图像作为主导源，并将深度图像、来自目标检测的说话者位置知识以及Gemini生成的语义描述作为补充源。随后，我们提出一种串行交互机制，以有效整合主导源与补充源。由此得到的多源知识会根据各源的各自贡献进行动态整合。这种多源空间知识的丰富交互与整合引导了语音生成模型，从而增强了沉浸式语音体验。实验结果表明，MS$^2$KU-VTTS在生成沉浸式语音方面超越了现有基线。演示和代码可在https://github.com/AI-S2-Lab/MS2KU-VTTS获取。
英文摘要：
Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize reverberant speech for the spoken content. Previous works focus on the RGB modality for global environmental modeling, overlooking the potential of multi-source spatial knowledge like depth, speaker position, and environmental semantics. To address these issues, we propose a novel multi-source spatial knowledge understanding scheme for immersive VTTS, termed MS2KU-VTTS. Specifically, we first prioritize RGB image as the dominant source and consider depth image, speaker position knowledge from object detection, and Gemini-generated semantic captions as supplementary sources. Afterwards, we propose a serial interaction mechanism to effectively integrate both dominant and supplementary sources. The resulting multi-source knowledge is dynamically integrated based on the respective contributions of each source.This enriched interaction and integration of multi-source spatial knowledge guides the speech generation model, enhancing the immersive speech experience. Experimental results demonstrate that the MS$^2$KU-VTTS surpasses existing baselines in generating immersive speech. Demos and code are available at: https://github.com/AI-S2-Lab/MS2KU-VTTS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
