口语语言模型在情感不一致语音上的情感识别能力评估
Evaluating Emotion Recognition in Spoken Language Models on Emotionally Incongruent Speech
AI总结：
该研究针对口语语言模型的多模态整合能力问题，用情感不一致语音数据集评估4个SLM的语音情感识别，发现其更依赖文本语义，并公开了代码与EMIS数据集。
摘要：
口语语言处理的进步推动了口语语言模型（SLMs）的发展，这类模型旨在通过联合学习文本和音频表示以实现通用音频理解，从而完成广泛的任务。尽管已取得可观的成果，但学界对这些模型的泛化能力、以及它们在内部表示中真正整合音频与文本模态的程度展开了越来越多的讨论。本研究使用情感不一致语音样本数据集，在语音情感识别任务上评估了4个SLM；该数据集的设定是：口语 utterance 的语义内容传达一种情绪，而语音表现力传达另一种情绪。结果表明，SLM 执行该任务时主要依赖文本语义而非语音情感，说明文本相关表示在很大程度上主导了声学表示。我们向社区公开了代码以及情感不一致合成语音数据集（EMIS）。
英文摘要：
Advancements in spoken language processing have driven the development of spoken language models (SLMs), designed to achieve universal audio understanding by jointly learning text and audio representations for a wide range of tasks. Although promising results have been achieved, there is growing discussion regarding these models' generalization capabilities and the extent to which they truly integrate audio and text modalities in their internal representations. In this work, we evaluate four SLMs on the task of speech emotion recognition using a dataset of emotionally incongruent speech samples, a condition under which the semantic content of the spoken utterance conveys one emotion while speech expressiveness conveys another. Our results indicate that SLMs rely predominantly on textual semantics rather than speech emotion to perform the task, indicating that text-related representations largely dominate over acoustic representations. We release both the code and the Emotionally Incongruent Synthetic Speech dataset (EMIS) to the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
