情感识别本质上具有模糊性,不确定性来源于评分者之间的分歧以及不同模态(如语音和文本)之间的差异。近年来,利用标签分布建模评分者模糊性引起了广泛关注。然而,模态模糊性仍处于探索阶段,多模态方法通常依赖于简单的特征融合,而没有明确处理模态之间的冲突。在本工作中,我们提出AmbER$^2$,一个双重模糊感知框架,通过具有分布性训练目标的教师-学生架构,同时建模评分者级别和模态级别模糊性。在IEMOCAP和MSP-Podcast上的评估表明,AmbER$^2$在传统交叉熵基线上始终提高了分布忠实度,并且在性能上与或优于最近的最先进系统相竞争。例如,在IEMOCAP上,AmbER$^2$在Bhattacharyya系数上实现了20.3%的相对提升(0.83 vs. 0.69),在R$^2$上实现了13.6%的提升(0.67 vs. 0.59),在准确率上实现了3.8%的提升(0.683 vs. 0.658),在F1上实现了4.5%的提升(0.675 vs. 0.646)。进一步分析不同模糊级别显示,显式建模模糊性对高不确定样本特别有益。这些发现突显了在构建稳健的情感识别系统时共同处理评分者和模态模糊性的重要性。
英文摘要:
Emotion recognition is inherently ambiguous, with uncertainty arising both from rater disagreement and from discrepancies across modalities such as speech and text. There is growing interest in modeling rater ambiguity using label distributions. However, modality ambiguity remains underexplored, and multimodal approaches often rely on simple feature fusion without explicitly addressing conflicts between modalities. In this work, we propose AmbER$^2$, a dual ambiguity-aware framework that simultaneously models rater-level and modality-level ambiguity through a teacher-student architecture with a distribution-wise training objective. Evaluations on IEMOCAP and MSP-Podcast show that AmbER$^2$ consistently improves distributional fidelity over conventional cross-entropy baselines and achieves performance competitive with, or superior to, recent state-of-the-art systems. For example, on IEMOCAP, AmbER$^2$ achieves relative improvements of 20.3% on Bhattacharyya coefficient (0.83 vs. 0.69), 13.6% on R$^2$ (0.67 vs. 0.59), 3.8% on accuracy (0.683 vs. 0.658), and 4.5% on F1 (0.675 vs. 0.646). Further analysis across ambiguity levels shows that explicitly modeling ambiguity is particularly beneficial for highly uncertain samples. These findings highlight the importance of jointly addressing rater and modality ambiguity when building robust emotion recognition systems.
A Study on the Ambiguity in Human Annotation of German Oral History Interviews for Perceived Emotion Recognition and Sentiment Analysis
Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS)(弗劳恩霍夫智能分析与信息系统研究所);Haus der Geschichte der Bundesrepublik Deutschland Foundation (HdG)(德意志联邦共和国历史之家基金会);University of Applied Sciences Bonn-Rhein-Sieg(波恩-莱茵-锡格应用技术大学);Autonomous Intelligent Systems (AIS), Computer Science Institute VI, Univ. of Bonn(波恩大学计算机科学第六研究所自主智能系统)
Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models
University of Melbourne(墨尔本大学);University of Auckland(奥克兰大学);South China Normal University(华南师范大学);MIT(麻省理工学院);University of Cambridge(剑桥大学);RMIT(皇家墨尔本理工大学)
VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Department of Computer Science, University of Sheffield, United Kingdom(英国谢菲尔德大学计算机科学系);Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA(美国南加州大学电气与计算机工程系信号分析与解释实验室(SAIL))
SuZhou Automotive Research Institute of Tsinghua University(清华大学苏州汽车研究院);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系);Hyundai Motor Advanced Tech. R&D Center;School of Vehicle and Mobility, Tsinghua University(清华大学车辆与移动系统学院)
Mandarin Humorous Homophone Recognition and Disambiguation in Automatic Speech Recognition
blind Anonymous to ISCSLP 1 School of Computer Science \& Engineering, UNSW Sydney, Australia 2 School of Electrical Engineering \& Telecommunications, UNSW Sydney, Australia