用于情感对话的双重信息语音语言模型
Dual Information Speech Language Models for Emotional Conversations
AI总结：
针对语音语言模型在情感对话中难以捕捉副语言信息且上下文理解下降的问题，提出双异构适配器与弱监督训练策略，解耦语言与副语言信息，实现高效的上下文情感理解。
摘要：
依赖基于文本的大型语言模型（LLMs）的对话系统常常忽略副语言线索，而这些线索对于理解情绪和意图至关重要。以语音作为输入的语音语言模型（SLMs）正成为一种极具前景的解决方案。然而，通过扩展冻结的LLMs构建的SLMs难以捕捉副语言信息，并表现出上下文理解能力的下降。我们将信息纠缠与不当的训练策略识别为关键问题。为了解决这些问题，我们提出了两个异构适配器，并建议采用弱监督训练策略。我们的方法解开了副语言与语言信息，使SLMs能够通过结构化表示来解释语音。该方法还通过受控随机性避免生成任务特定向量，从而保留了上下文理解能力。此方法仅在常见数据集上训练适配器，确保了参数与数据效率。实验表明，该模型在情感对话任务中展现出极具竞争力的性能，证明了其能够在上下文环境中有效整合副语言与语言信息的能力。
英文摘要：
Conversational systems relying on text-based large language models (LLMs) often overlook paralinguistic cues, essential for understanding emotions and intentions. Speech-language models (SLMs), which use speech as input, are emerging as a promising solution. However, SLMs built by extending frozen LLMs struggle to capture paralinguistic information and exhibit reduced context understanding. We identify entangled information and improper training strategies as key issues. To address these issues, we propose two heterogeneous adapters and suggest a weakly supervised training strategy. Our approach disentangles paralinguistic and linguistic information, enabling SLMs to interpret speech through structured representations. It also preserves contextual understanding by avoiding the generation of task-specific vectors through controlled randomness. This approach trains only the adapters on common datasets, ensuring parameter and data efficiency. Experiments demonstrate competitive performance in emotional conversation tasks, showcasing the model's ability to effectively integrate both paralinguistic and linguistic information within contextual settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
