arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05007cs.SDcs.AIeess.AS
收藏

用于多模态情感分析的音频引导融合技术

Audio-Guided Fusion Techniques for Multimodal Emotion Analysis

Beijing University of Posts and Telecommunications(北京邮电大学);State Key Laboratory of Networking and Switching Technology(网络与交换技术国家重点实验室);School of Cyberspace Security(网络空间安全学院)

Pujin Shi, Fei Gao

AI总结:

针对MER2024半监督情感识别赛道,本文通过微调CLIP-vit-large与Baichuan-13B、提出AGT音频引导融合、伪标签自训练及先验投票机制,获得赛道第三名。

摘要:

本文提出了一种面向MER2024半监督学习赛道(MER-SEMI)的解决方案。首先,为提升特征提取器在情感分类任务上的性能,我们利用标注数据对视频和文本特征提取器(具体为CLIP-vit-large和Baichuan-13B)进行了微调,该方法有效保留了视频所传达的原始情感信息。其次,我们提出了一种音频引导Transformer(AGT)融合机制,该机制利用Hubert-large的鲁棒性,在融合通道间与通道内信息方面表现出卓越效果。第三,为提升模型准确率,我们通过将高置信度未标注数据用作伪标签,迭代式地应用自监督学习。最后,通过黑盒探测,我们发现训练集与测试集之间存在不平衡的数据分布,因此采用了基于先验知识的投票机制。结果证明了我们策略的有效性,最终在MER-SEMI赛道中获得第三名。

英文摘要:

In this paper, we propose a solution for the semi-supervised learning track (MER-SEMI) in MER2024. First, in order to enhance the performance of the feature extractor on sentiment classification tasks,we fine-tuned video and text feature extractors, specifically CLIP-vit-large and Baichuan-13B, using labeled data. This approach effectively preserves the original emotional information conveyed in the videos. Second, we propose an Audio-Guided Transformer (AGT) fusion mechanism, which leverages the robustness of Hubert-large, showing superior effectiveness in fusing both inter-channel and intra-channel information. Third, To enhance the accuracy of the model, we iteratively apply self-supervised learning by using high-confidence unlabeled data as pseudo-labels. Finally, through black-box probing, we discovered an imbalanced data distribution between the training and test sets. Therefore, We adopt a prior-knowledge-based voting mechanism. The results demonstrate the effectiveness of our strategy, ultimately earning us third place in the MER-SEMI track.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑