arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.04585eess.AS
收藏

UniTalker:对话式语音-视觉合成

UniTalker: Conversational Speech-Visual Synthesis

Yifan Hu, Rui Liu, Yi Ren, Xiang Yin, Haizhou Li

AI总结:

针对现有对话语音合成仅感知文本语音且仅输出语音的局限,提出UniTalker统一模型,利用多模态上下文生成共情语音与自然说话人脸动画,实现连贯视听回应。

摘要:

对话式语音合成(CSS)是用户与智能体交互领域的一项关键任务,旨在为用户生成更具表现力和共情能力的语音。然而,众所周知,在现实世界的人际交流中,“倾听”和“眼神接触”在传达情感方面起着至关重要的作用。现有的CSS研究仅限于感知对话上下文中的文本和语音,这限制了其有效性。此外,仅语音的回应进一步限制了交互体验。为解决这些局限,我们引入对话式语音-视觉合成(CSVS)任务,作为传统CSS的扩展。通过利用多模态对话上下文,它为用户提供连贯的视听回应。为此,我们开发了一个名为UniTalker的CSVS系统,这是一个统一模型,无缝集成了多模态感知和多模态渲染能力。具体而言,它利用大规模语言模型全面理解对话上下文中的多模态线索,包括说话人、文本、语音和说话人脸动画。之后,它采用多任务序列预测,首先推断目标话语的情感,然后生成共情语音和自然的说话人脸动画。为确保生成的语音-视觉内容在情感、内容和时长方面保持一致,我们引入了三项关键优化:1)设计专门的神经地标编解码器,用于对表情序列进行分词和重建;2)提出双模态语音-视觉硬对齐解码策略;3)在生成阶段应用情感引导渲染。全面的客观和主观实验表明,我们的模型能合成更具共情性的语音,并为用户提供更自然、情感一致的说话人脸动画。

英文摘要:

Conversational Speech Synthesis (CSS) is a key task in the user-agent interaction area, aiming to generate more expressive and empathetic speech for users. However, it is well-known that "listening" and "eye contact" play crucial roles in conveying emotions during real-world interpersonal communication. Existing CSS research is limited to perceiving only text and speech within the dialogue context, which restricts its effectiveness. Moreover, speech-only responses further constrain the interactive experience. To address these limitations, we introduce a Conversational Speech-Visual Synthesis (CSVS) task as an extension of traditional CSS. By leveraging multimodal dialogue context, it provides users with coherent audiovisual responses. To this end, we develop a CSVS system named UniTalker, which is a unified model that seamlessly integrates multimodal perception and multimodal rendering capabilities. Specifically, it leverages a large-scale language model to comprehensively understand multimodal cues in the dialogue context, including speaker, text, speech, and the talking-face animations. After that, it employs multi-task sequence prediction to first infer the target utterance's emotion and then generate empathetic speech and natural talking-face animations. To ensure that the generated speech-visual content remains consistent in terms of emotion, content, and duration, we introduce three key optimizations: 1) Designing a specialized neural landmark codec to tokenize and reconstruct facial expression sequences. 2) Proposing a bimodal speech-visual hard alignment decoding strategy. 3) Applying emotion-guided rendering during the generation stage. Comprehensive objective and subjective experiments demonstrate that our model synthesizes more empathetic speech and provides users with more natural and emotionally consistent talking-face animations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑