arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.14221eess.AScs.SD
收藏

强独立,更强联合:通过最优传输协同模态绑定基础模型以进行非语言情感识别

Strong Alone, Stronger Together: Synergizing Modality-Binding Foundation Models with Optimal Transport for Non-Verbal Emotion Recognition

IIIT-Delhi(德里印度信息技术学院);UPES(佩鲁梅技术与科学大学);Reliance Jio AICoE(信实焦人工智能卓越中心);University of Tartu(塔尔图大学);IIT-Dharwad(达尔瓦德印度理工学院);IIIT-Dharwad(达尔瓦德印度信息技术学院)

Orchid Chetia Phukan, Mohd Mujtaba Akhtar, Girish, Swarup Ranjan Behera, Sishir Kalita, Arun Balaji Buduru, Rajesh Sharma, S. R Mahadeva Prasanna

AI总结:

本研究提出MATA框架,通过最优传输协同多模态基础模型(如LanguageBind和ImageBind)的表示,在非语言声音情感识别任务中优于单一模型与基线融合方法,实现SOTA性能。

摘要:

在本研究中,我们探讨了多模态基础模型(MFMs)在非语言声音情感识别中的应用。我们假设,MFMs凭借跨多模态的联合预训练,能更好地解释和区分仅在音频基础模型(AFMs)中可能显得模糊的细微情感线索,从而在非语言声音情感识别(NVER)中更为有效。为验证假设,我们从最先进(SOTA)的MFMs和AFMs中提取表示,并在基准NVER数据集上进行评估。受语音识别和音频深度伪造检测研究的启发,我们还探讨了结合精选基础模型表示以进一步增强NVER的潜力。为此,我们提出了名为MATA(通过传输注意力进行模态内对齐)的框架。通过MATA结合MFMs:LanguageBind和ImageBind,我们在ASVP-ESD、JNV和VIVAE数据集上取得了最高性能,准确率分别为76.47%、77.40%、75.12%,F1分数分别为70.35%、76.19%、74.63%,优于单个FM和基线融合技术,并在基准数据集上报告了SOTA。

英文摘要:

In this study, we investigate multimodal foundation models (MFMs) for emotion recognition from non-verbal sounds. We hypothesize that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs). To validate our hypothesis, we extract representations from state-of-the-art (SOTA) MFMs and AFMs and evaluated them on benchmark NVER datasets. We also investigate the potential of combining selected foundation model representations to enhance NVER further inspired by research in speech recognition and audio deepfake detection. To achieve this, we propose a framework called MATA (Intra-Modality Alignment through Transport Attention). Through MATA coupled with the combination of MFMs: LanguageBind and ImageBind, we report the topmost performance with accuracies of 76.47%, 77.40%, 75.12% and F1-scores of 70.35%, 76.19%, 74.63% for ASVP-ESD, JNV, and VIVAE datasets against individual FMs and baseline fusion techniques and report SOTA on the benchmark datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑