副语言增强的口语对话大语言建模
Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
AI总结：
本文提出ParalinGPT，通过序列化多任务多模态框架融合文本、语音嵌入和副语言属性，在Switchboard-1上显著提升口语对话的情感预测与响应生成性能。
摘要：
大语言模型（LLMs）在聊天、推理和问答等任务中展现了卓越的能力。然而，标准LLMs可能忽略关键的副语言信息，如情感、情绪和说话风格，这些信息对于实现自然、类人的口语对话至关重要，尤其是当这些信息通过声学线索传递时。因此，我们提出了副语言增强的生成式预训练Transformer（ParalinGPT），这是一种利用文本和语音模态来更好地建模口语对话的语言内容和副语言属性的LLM。该模型将对话上下文的文本、语音嵌入和副语言属性作为输入提示，纳入一个序列化多任务多模态框架中。具体而言，我们的框架按照当前副语言属性预测、响应副语言属性预测和响应文本生成的顺序对任务进行序列化，并采用自回归条件建模。我们使用Switchboard-1语料库（包括其情感标签作为副语言属性）作为口语对话数据集。实验结果表明，所提出的序列化多任务方法在当前和响应情感分类上优于典型的序列分类技术。此外，利用对话上下文和语音嵌入显著改善了响应文本生成和情感预测。我们提出的框架在当前情感准确率、响应情感准确率和响应文本BLEU分数上分别实现了6.7%、12.0%和3.5%的相对提升。
英文摘要：
Large Language Models (LLMs) have demonstrated superior abilities in tasks such as chatting, reasoning, and question-answering. However, standard LLMs may ignore crucial paralinguistic information, such as sentiment, emotion, and speaking style, which are essential for achieving natural, human-like spoken conversation, especially when such information is conveyed by acoustic cues. We therefore propose Paralinguistics-enhanced Generative Pretrained Transformer (ParalinGPT), an LLM that utilizes text and speech modalities to better model the linguistic content and paralinguistic attributes of spoken dialogue. The model takes the conversational context of text, speech embeddings, and paralinguistic attributes as input prompts within a serialized multitasking multimodal framework. Specifically, our framework serializes tasks in the order of current paralinguistic attribute prediction, response paralinguistic attribute prediction, and response text generation with autoregressive conditioning. We utilize the Switchboard-1 corpus, including its sentiment labels as the paralinguistic attribute, as our spoken dialogue dataset. Experimental results indicate the proposed serialized multitasking method outperforms typical sequence classification techniques on current and response sentiment classification. Furthermore, leveraging conversational context and speech embeddings significantly improves both response text generation and sentiment prediction. Our proposed framework achieves relative improvements of 6.7%, 12.0%, and 3.5% in current sentiment accuracy, response sentiment accuracy, and response text BLEU score, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
