arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.19258cs.CL
收藏

VisualSpeech:利用视频增强 TTS 中的韵律建模

VisualSpeech: Enhancing Prosody Modeling in TTS Using Video

Department of Computer Science, The University of Sheffield, United Kingdom(计算机科学系,谢菲尔德大学,英国)

Shumin Que, Anton Ragni

AI总结:

本文提出 VisualSpeech 模型,通过融合视频视觉信息与文本信息来改进 TTS 韵律预测,实验证明视觉特征能提升合成语音的表现力。

摘要:

文本到语音(TTS)合成面临一个固有挑战:给定单个文本输入时,需要生成具有不同韵律的多个语音输出。尽管先前研究已通过从文本和语音中预测韵律信息来解决这一问题,但视频等额外上下文信息在许多应用中虽然可用,却仍未得到充分利用。本文研究了整合视觉上下文以增强韵律预测的潜力。我们提出了一种新模型 VisualSpeech,该模型结合视觉和文本信息来改进 TTS 中的韵律生成。实验结果表明,引入视觉特征能够改善韵律建模,从而增强合成语音的表现力。音频样本可在 https://ariameetgit.github.io/VISUALSPEECH-SAMPLES/ 获取。

英文摘要:

Text-to-Speech (TTS) synthesis faces the inherent challenge of producing multiple speech outputs with varying prosody given a single text input. While previous research has addressed this by predicting prosodic information from both text and speech, additional contextual information, such as video, remains under-utilized despite being available in many applications. This paper investigates the potential of integrating visual context to enhance prosody prediction. We propose a novel model, VisualSpeech, which incorporates visual and textual information for improving prosody generation in TTS. Empirical results indicate that incorporating visual features improves prosodic modeling, enhancing the expressiveness of the synthesized speech. Audio samples are available at https://ariameetgit.github.io/VISUALSPEECH-SAMPLES/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑