I2TTS: 具备空间感知的图像指示沉浸式文本转语音合成
I2TTS: Image-indicated Immersive Text-to-speech Synthesis with Spatial Perception
AI总结：
针对现有TTS忽略空间感知的问题，提出I2TTS多模态方法，通过场景提示编码器和混响分类优化技术，在保证语音自然度的同时实现高质量场景空间匹配，推动上下文感知语音合成发展。
摘要：
控制语音合成的风格与特征对于让输出适配特定场景及用户需求至关重要。以往的文本转语音（TTS）研究主要聚焦于生成自然语音的技术层面，例如语调、节奏与清晰度。然而，这些研究忽略了当前人们对合成语音的空间感知关注度日益提升的事实，这种空间感知可在游戏与虚拟现实中提供沉浸式体验。为解决该问题，本文提出一种新型多模态TTS方法，即图像指示沉浸式文本转语音合成（I2TTS）。具体而言，我们引入了场景提示编码器，将视觉场景提示直接整合到合成流程中，以控制语音生成过程。此外，我们提出了混响分类与优化技术，对合成的梅尔频谱图进行调整以增强沉浸式体验，确保涉及的混响条件与场景精准匹配。实验结果表明，我们的模型在不损失语音自然度的前提下实现了高质量的场景与空间匹配，标志着上下文感知语音合成领域取得了重大进展。项目演示页面：https://spatialTTS.github.io/ 关键词：语音合成、场景提示、空间感知
英文摘要：
Controlling the style and characteristics of speech synthesis is crucial for adapting the output to specific contexts and user requirements. Previous Text-to-speech (TTS) works have focused primarily on the technical aspects of producing natural-sounding speech, such as intonation, rhythm, and clarity. However, they overlook the fact that there is a growing emphasis on spatial perception of synthesized speech, which may provide immersive experience in gaming and virtual reality. To solve this issue, in this paper, we present a novel multi-modal TTS approach, namely Image-indicated Immersive Text-to-speech Synthesis (I2TTS). Specifically, we introduce a scene prompt encoder that integrates visual scene prompts directly into the synthesis pipeline to control the speech generation process. Additionally, we propose a reverberation classification and refinement technique that adjusts the synthesized mel-spectrogram to enhance the immersive experience, ensuring that the involved reverberation condition matches the scene accurately. Experimental results demonstrate that our model achieves high-quality scene and spatial matching without compromising speech naturalness, marking a significant advancement in the field of context-aware speech synthesis. Project demo page: https://spatialTTS.github.io/ Index Terms-Speech synthesis, scene prompt, spatial perception
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
