We used two multimodal models for continuous valence-arousal recognition using visual, audio, and linguistic information. The first model is the same as we used in ABAW2 and ABAW3, which employs the leader-follower attention. The second model has the same architecture for spatial and temporal encoding. As for the fusion block, it employs a compact and straightforward channel attention, borrowed from the End2You toolkit. Unlike our previous attempts that use Vggish feature directly as the audio feature, this time we feed the pre-trained VGG model using logmel-spectrogram and finetune it during the training. To make full use of the data and alleviate over-fitting, cross-validation is carried out. The code is available at https://github.com/sucv/ABAW3.
Sync-TVA: A Graph-Attention Framework for Multimodal Emotion Recognition with Cross-Modal Fusion
James Watt School of Engineering, University of Glasgow(格拉斯哥大学詹姆斯·瓦特工程学院);University of Bristol(布里斯托大学);School of Engineering Mathematics and Technology, University of Bristol(布里斯托大学工程数学与技术学院);School of Computing Science, University of Glasgow(格拉斯哥大学计算科学学院);Department of Civil, Environmental & Geomatic Engineering, University College London (UCL)(伦敦大学学院(UCL)土木、环境与测绘工程系)
College of Computer and Mathematics, Central South University of Forestry and Technology(计算机与数学学院,中央南林业科技大学);Department of Computer Science, State University of New York(计算机科学系,纽约州立大学)
Orthogonal Disentanglement with Projected Feature Alignment for Multimodal Emotion Recognition in Conversation
School of Computer Science Sichuan University(四川大学计算机学院);Faculty of Computing Harbin Institute of Technology(哈尔滨工业大学计算学部);College of Computer Science and Technology Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院)
SEER: Semantic Enhancement and Emotional Reasoning Network for Multimodal Fake News Detection
School of Artificial Intelligence, Optics, and Electronics (iOPEN), Northwestern Polytechnical University(人工智能、光学与电子学院(iOPEN),西北工业大学);School of Computer Science, Northwestern Polytechnical University(计算机学院,西北工业大学);Unit 93212 of People’s Liberation Army of China(中国人民解放军第九三二一二单位);School of Marine Science and Technology, Northwestern Polytechnical University(海洋科学与技术学院,西北工业大学);Cyberspace Institute of Advanced Technology, Guangzhou University(高级技术网络空间研究院,广州大学)
Multimodal Emotion Recognition with Large Language Models
Tsinghua University(清华大学);Harbin Institute of Technology, Weihai(哈尔滨工业大学(威海));Academy of Cyber(网络空间安全学院);College of Computer Science, Nankai University(南开大学计算机科学学院)
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University(香港理工大学电子与电气工程系);Speech, Language, and Cognition Laboratory, The University of Hong Kong(香港大学语音、语言与认知实验室)
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Shenzhen Technology University(深圳技术大学);Carnegie Mellon University(卡内基梅隆大学);Alibaba Group(阿里巴巴集团);National University of Singapore(新加坡国立大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Noise-Resistant Multimodal Transformer for Emotion Recognition
China University of Geosciences (Wuhan)(中国地质大学(武汉));JD Explore Academy, JD.com(京东集团京东探索研究院);Australian Catholic University(澳大利亚天主教大学);The University of Sydney(悉尼大学)
XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);th Research Institute, China Electronics Technology Group Corporation(中国电子科技集团第五十四研究所)
本文讨论了情感分类的问题。该任务包括预测最能描述短视频剪辑中包含的情绪的情绪标签(从一组可能的标签中选取)。建立在一个标准的框架-躺在描述视频的音频和视觉特征使用的监督分类器来推断标签-本文研究了几个新的方向。首先,提出了基于二维和三维卷积神经网络的改进人脸描述子。其次,本文探讨了几种融合方法,时间和多模态,包括一种新的分层方法相结合的功能和分数。此外,我们仔细审查了流水线的不同阶段,并设计了一个适应任务的CNN架构;这一点很重要,因为与问题的难度相比,训练集的大小很小,使得泛化困难。该模型在2017年的Emotion in the Wild挑战赛中排名第四,准确率为58.8%。