Training a multimodal network is challenging and it requires complex architectures to achieve reasonable performance. We show that one reason for this phenomena is the difference between the convergence rate of various modalities. We address this by pre-training modality-specific sub-networks in multimodal architectures independently before end-to-end training of the entire network. Furthermore, we show that the addition of an attention mechanism between sub-networks after pre-training helps identify the most important modality during ambiguous scenarios boosting the performance. We demonstrate that by performing these two tricks a simple network can achieve similar performance to a complicated architecture that is significantly more expensive to train on multiple tasks including sentiment analysis, emotion recognition, and speaker trait recognition.
CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis
Information Networking Institute at Carnegie Mellon University (CMU)(卡内基梅隆大学信息网络研究所);School of Computer Science and Technology at Tongji University(同济大学计算机科学与技术学院)
Boosting Multimodal Federated Learning via Chained Modality Optimization
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China(天津理工大学计算机科学与工程学院);Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所);College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China(内蒙古大学计算机学院)
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
Computational Biomedicine Laboratory of the Foundation for Research and Technology (FORTH)(希腊国家研究与技术基金会计算生物医学实验室);Department of Electrical & Computer Engineering Hellenic Mediterranean University(希腊地中海大学电气与计算机工程系)
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China(清华大学软件学院,北京,中国);TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院,中国);School of New Media and Communication, Tianjin University, China(天津大学新媒体与传播学院,中国)
Improving Multimodal Learning Balance and Sufficiency through Data Remixing
School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及交叉应用重点实验室);College of Computer Science, Beijing University of Technology, Beijing, China(北京理工大学计算机学院)
Deep Multimodal Learning with Missing Modality: A Survey
The Australian National University(澳大利亚国立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Adelaide University(阿德莱德大学);The University of Surrey(萨里大学)
Rebalanced Multimodal Learning with Data-aware Unimodal Sampling
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Computer Science and Technology, Zhejiang Sci-Tech University(浙江理工大学计算机科学与技术学院);School of Computer Science and Communication Engineering, Jiangsu University(江苏大学计算机科学与通信工程学院)
Enhancing multimodal cooperation via sample-level modality valuation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
College of Computer and Mathematics, Central South University of Forestry and Technology(计算机与数学学院,中央南林业科技大学);Department of Computer Science, State University of New York(计算机科学系,纽约州立大学)
Robust Multimodal Large Language Models Against Modality Conflict
School of Artificial Intelligence(人工智能学院);Data Science, University of Science(数据科学,科学大学);Department of Electronic Engineering(电子工程系);Information Science, University of Science(信息科学,科学大学);Huawei Technologies Co., Ltd.(华为技术有限公司)