通过生成引人入胜的语音迈向类人多模态对话智能体
Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech
AI总结：
本文提出一种类人多模态对话智能体，通过构建MultiSensory Conversation语音数据集并基于多模态大语言模型生成文本回复与语音描述，结合视觉和音频模态生成自然且引人入胜的语音回复。
摘要：
人类对话涉及语言、语音和视觉线索，每种媒介都提供互补信息。例如，语音传达了一种仅靠文本无法完全捕捉的情绪或语气。虽然多模态大语言模型专注于从多样化输入生成文本回复，但对生成自然且引人入胜的语音关注较少。我们提出一种类人智能体，根据对话情绪和响应风格信息生成语音回复。为实现这一目标，我们构建了一个以语音为重点的新型 MultiSensory Conversation 数据集，使智能体能够生成自然语音。随后，我们提出一种基于多模态大语言模型的模型，用于生成文本回复和语音描述，并利用这些描述生成涵盖副语言信息的语音。实验结果表明，在对话中同时利用视觉和音频模态能够有效生成引人入胜的语音。源代码可在 https://github.com/kimtaesu24/MSenC 获取。
英文摘要：
Human conversation involves language, speech, and visual cues, with each medium providing complementary information. For instance, speech conveys a vibe or tone not fully captured by text alone. While multimodal LLMs focus on generating text responses from diverse inputs, less attention has been paid to generating natural and engaging speech. We propose a human-like agent that generates speech responses based on conversation mood and responsive style information. To achieve this, we build a novel MultiSensory Conversation dataset focused on speech to enable agents to generate natural speech. We then propose a multimodal LLM-based model for generating text responses and voice descriptions, which are used to generate speech covering paralinguistic information. Experimental results demonstrate the effectiveness of utilizing both visual and audio modalities in conversation to generate engaging speech. The source code is available in https://github.com/kimtaesu24/MSenC
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
