Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data. This research area's major concern lies in developing an extraordinary fusion scheme that can extract and integrate key information from various modalities. However, one issue that may restrict previous work to achieve a higher level is the lack of proper modeling for the dynamics of the competition between the independence and relevance among modalities, which could deteriorate fusion outcomes by causing the collapse of modality-specific feature space or introducing extra noise. To mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel end-to-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. The two parts are trained simultaneously such that the combat between them is simulated. The model takes two bimodal pairs as input due to the known information imbalance among modalities. In addition, we leverage a gated control mechanism in the Transformer architecture to further improve the final output. Experimental results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the SOTA. The implementation of this work is available at https://github.com/declare-lab/multimodal-deep-learning.
CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis
Information Networking Institute at Carnegie Mellon University (CMU)(卡内基梅隆大学信息网络研究所);School of Computer Science and Technology at Tongji University(同济大学计算机科学与技术学院)
AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba
Pengcheng Laboratory(鹏城实验室);Shaanxi University of Science & Technology(陕西科技大学);School of Computer Science(计算机科学学院);Northwestern Polytechnical University(西北工业大学)
Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach
College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算机与数据科学学院);Singapore University of Social Sciences, Singapore(新加坡社会科学研究大学)
MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
School of Computer Science and Technology, Xinjiang University(新疆大学计算机科学与技术学院);Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing(丝绸之路多语言认知计算国际联合研究实验室);Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center(新疆多模态智能处理与信息安全工程技术研究中心);Pengcheng Laboratory Xinjiang Network Node(鹏城实验室新疆网络节点);Embodied Intelligence Joint Laboratory(具身智能联合实验室)
Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis
University of Chinese Academy of Sciences(中国科学院大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Henan University(河南大学)
Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));China University of Geosciences, Wuhan(中国地质大学(武汉));Nanyang Technological University(南洋理工大学);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院)
Towards Robust Multimodal Sentiment Analysis with Incomplete Data
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);University College London(伦敦大学学院);School of Computer Science, Wuhan University(武汉大学计算机学院)
FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis
Shanghai University(上海大学);North China Electric Power University(华北电力大学);Université de Technologie de Compiègne(法国图卢兹国立理工学院);Nanyang Technological University(南洋理工大学);City University of Hong Kong(香港城市大学);Macquarie University(麦考瑞大学)
RNG: Reducing Multi-level Noise and Multi-grained Semantic Gap for Joint Multimodal Aspect-Sentiment Analysis
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, UCAS(中国科学院大学网络空间安全学院);Academy of Cyber(网络空间研究院)
M-SENA: An Integrated Platform for Multimodal Sentiment Analysis
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Hebei University of Science and Technology(河北科技大学)
Variational Fusion for Multimodal Sentiment Analysis
A*STAR AI Initiative, Institute of High Performance Computing(A*STAR人工智能计划、高性能计算研究院);Adobe Research(奥多比研究院);University of Michigan(密歇根大学);Centro de Investigación en Computación, Instituto Politécnico Nacional(国立理工学院计算研究中心);SUTD(新加坡科技设计大学)
ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors
Southwest Jiaotong University(西南交通大学);Microsoft Research Asia(微软亚洲研究院);Southwestern University of Finance and Economics(西南财经大学);Ocean University of China(中国海洋大学);Tencent(腾讯)