提升大语言模型在语音对话中捕捉多样化说话风格并做出恰当响应的能力
Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations
AI总结：
本文提出Spoken-LLM框架，通过构建语音到语音数据集StyleTalk并采用两阶段训练流程，使大语言模型能够理解说话风格并做出恰当响应，实验证明其优于纯文本基线和现有语音LLM方法。
摘要：
在语音对话中，即使两个当前轮次是相同的句子，当它们以不同的风格说出时，其响应可能仍然不同。语音风格包含副语言和韵律信息，标志着文本与语音模态之间最显著的差异。当使用纯文本LLM对语音对话进行建模时，纯文本LLM无法根据当前轮次的说话风格给出不同的响应。在本文中，我们专注于使LLM能够倾听说话风格并做出恰当响应。我们的目标是教会LLM“即使句子相同，如果以不同的风格说出，其对应的响应可能也不同”。由于没有合适的数据集来实现这一目标，我们收集了一个语音到语音数据集StyleTalk，其具有以下期望特征：当两个当前语音具有相同内容但以不同风格说出时，它们的响应将不同。为了教会LLM理解并对说话风格做出恰当响应，我们提出了Spoken-LLM框架，该框架能够对语言内容和说话风格进行建模。我们使用StyleTalk数据集训练Spoken-LLM，并设计了一个两阶段训练流程，以帮助Spoken-LLM更好地学习说话风格。基于大量实验，我们表明Spoken-LLM优于纯文本基线和先前的语音LLM方法。
英文摘要：
In spoken dialogue, even if two current turns are the same sentence, their responses might still differ when they are spoken in different styles. The spoken styles, containing paralinguistic and prosodic information, mark the most significant difference between text and speech modality. When using text-only LLMs to model spoken dialogue, text-only LLMs cannot give different responses based on the speaking style of the current turn. In this paper, we focus on enabling LLMs to listen to the speaking styles and respond properly. Our goal is to teach the LLM that "even if the sentences are identical if they are spoken in different styles, their corresponding responses might be different". Since there is no suitable dataset for achieving this goal, we collect a speech-to-speech dataset, StyleTalk, with the following desired characteristics: when two current speeches have the same content but are spoken in different styles, their responses will be different. To teach LLMs to understand and respond properly to the speaking styles, we propose the Spoken-LLM framework that can model the linguistic content and the speaking styles. We train Spoken-LLM using the StyleTalk dataset and devise a two-stage training pipeline to help the Spoken-LLM better learn the speaking styles. Based on extensive experiments, we show that Spoken-LLM outperforms text-only baselines and prior speech LLMs methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
