从语音到可编辑概念：用概念瓶颈模型探究情感识别
From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models
- University of Sheffield（谢菲尔德大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究将概念瓶颈模型应用于语音情感识别，通过实验揭示LLM对转录文本的偏差及概念移除对个体预测的显著影响，强调需关注细粒度可解释性。
AI中文摘要：
语音情感识别（SER）是为话语分配情感标签的任务。早期系统依赖声学特征，而近期方法结合多种模态，最常见的是语音和文本。然而，在许多数据集上性能仍然不佳。因此，大语言模型（LLMs）引起了SER领域的兴趣，因为它们可以结合指令联合处理多种输入。但是，直接音频输入引发了可解释性问题。为了解决图像分类中的类似问题，概念瓶颈模型被引入。本研究将概念瓶颈模型适配到SER，以检验个体预测如何依赖于转录文本、声学描述和说话人属性。实验在CREMA-D、IEMOCAP和MELD数据集上测试了三个LLM，概念由独立工具提取。在脚本化语料库上，零样本设置下LLM强烈偏向于转录文本，这使CREMA-D上的Macro-F1从27.8降至5.8。微调消除了这种偏差，转录文本将Macro-F1从41.8提升至45.1。移除语速在CREMA-D上将48%的Neutral预测变为Disgust；在MELD上移除强度级别改变了预测，尽管Macro-F1变化很小。这些发现表明，仅凭总体性能变化无法捕捉概念移除对个体预测的影响。
英文摘要：
Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.
