arXiv 2603.236732026-10-01eess.AScs.SD

Crab：多层对比监督以提升在表演和自然语音条件下的语音情感识别

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition

Dept. of Computer Engineering and Automation (DCA), Faculdade de Engenharia Elétrica e de Computação（计算机工程与自动化系（DCA）、电气与计算机工程学院）；AI Lab.（人工智能实验室）；Institute of Computing, Universidade Estadual de Campinas, UNICAMP（计算学院，坎皮纳斯州立大学，UNICAMP）

Lucas H. Ueda, João G. T. Lima, Paula D. P. Costa

备注：: Submitted to Speech Communication

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Crab模型，通过多层对比监督策略提升语音情感识别在表演和自然语音条件下的性能，采用跨模态Transformer架构和多任务对比学习，有效解决类别不平衡问题。

摘要：

语音情感识别（SER）在现实场景中仍面临严峻的类别不平衡和普遍存在自发自然语音的挑战。尽管近期方法利用自监督学习（SSL）表示和语音文本多模态融合，但大多数现有方法仅在最终分类层应用监督，限制了中间表示的判别能力。本文提出Crab（Contrastive Representation and Multimodal Aligned Bottleneck），一种双模态跨模态Transformer架构，整合WavLM的语音表示和RoBERTa的文本表示，并结合新颖的多层对比监督（MLCS）策略。MLCS在网络多层注入多正例对比学习信号，鼓励模型在整个过程中生成情感判别性表示，无需在推理时引入额外参数。为进一步解决数据不平衡问题，我们在训练中采用加权交叉熵。我们在三个涵盖不同情感自然程度的基准数据集上评估了所提出的方法：IEMOCAP、MELD和MSP-Podcast 2.0。实验结果表明，Crab在所有数据集上均优于强大的单模态和多模态基线，特别是在自然和高度不平衡条件下有显著提升。这些发现突显了多层对比监督作为SER通用且稳健策略的有效性。官方实现可在https://github.com/AI-Unicamp/Crab找到。

英文摘要：

Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel \textit{Multi Layer Contrastive Supervision} (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of \textit{Multi Layer Contrastive Supervision} as a general and robust strategy for SER. Official implementation can be found in https://github.com/AI-Unicamp/Crab.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。