Diagnosing and Re-learning for Balanced Multimodal Learning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Engineering Research Center of Next-Generation Search and Recommendation（下一代搜索与推荐技术工程研究中心）
To overcome the imbalanced multimodal learning problem, where models prefer the training of specific modalities, existing methods propose to control the training of uni-modal encoders from different perspectives, taking the inter-modal performance discrepancy as the basis. However, the intrinsic limitation of modality capacity is ignored. The scarcely informative modalities can be recognized as ``worse-learnt'' ones, which could force the model to memorize more noise, counterproductively affecting the multimodal model ability. Moreover, the current modality modulation methods narrowly concentrate on selected worse-learnt modalities, even suppressing the training of others. Hence, it is essential to consider the intrinsic limitation of modality capacity and take all modalities into account during balancing. To this end, we propose the Diagnosing \& Re-learning method. The learning state of each modality is firstly estimated based on the separability of its uni-modal representation space, and then used to softly re-initialize the corresponding uni-modal encoder. In this way, the over-emphasizing of scarcely informative modalities is avoided. In addition, encoders of worse-learnt modalities are enhanced, simultaneously avoiding the over-training of other modalities. Accordingly, multimodal learning is effectively balanced and enhanced. Experiments covering multiple types of modalities and multimodal frameworks demonstrate the superior performance of our simple-yet-effective method for balanced multimodal learning. The source code and dataset are available at \url{https://github.com/GeWu-Lab/Diagnosing_Relearning_ECCV2024}.
Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers
Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea；Department of Electrical；Computer Engineering, Seoul National University, Seoul, South Korea；Department of Computer Science \& Engineering, Korea University, Seoul, South Korea；ISRC, Seoul National University, Seoul, South Korea
TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
University of California, Los Angeles（加州大学洛杉矶分校）；National Taiwan University（国立台湾大学）；Texas A&M University（德克萨斯A&M大学）；NVIDIA AI Technology Center（英伟达AI技术中心）；Arena Intelligence Inc（Arena Intelligence公司）
Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection
Beijing University of Posts and Telecommunications（北京邮电大学）；Huawei Technologies Ltd.（华为技术有限公司）；Peking University（北京大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras
Bytedance（字节跳动）；Peking University（北京大学）；Peking University People’s Hospital（北京大学人民医院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The Chinese University of Hong Kong（香港中文大学）
Renmin University of China（中国人民大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, …
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
MICS, CentraleSupélec - Université Paris-Saclay, France（MICS，中央圣艾尔布兰大学-巴黎萨克雷大学，法国）；Google DeepMind, London, UK（谷歌DeepMind，伦敦，英国）；CONICET, Universidad de Buenos Aires, Argentina（CONICET，布宜诺斯艾利斯大学，阿根廷）
UC Santa Barbara（加州大学圣巴巴拉分校）；University of Adelaide（阿德莱德大学）；Google（谷歌）；UC Santa Cruz（加州大学圣克鲁兹分校）
Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the agents' inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we …
Multimodal Attention-Aware Fusion for Diagnosing Distal Myopathy: Evaluating Model Interpretability and Clinician Trust
Information Systems Group, Eindhoven University of Technology, The Netherlands（埃因霍温技术大学信息系统组）；Eindhoven Artificial Intelligence Systems Institute, The Netherlands（埃因霍温人工智能系统研究所）；Department of Computer Science and Technology, University of Cambridge, United Kingdom（剑桥大学计算机科学与技术系）；Department of Medicine - DIMED, Padua University Hospital, Italy（帕多瓦大学医院医学部-DIMED）；Department of Environmental Sciences, Informatics, and Statistics, Ca’ Foscari University of Venice, Italy（威尼斯卡弗里大学环境科学、信息学与统计学系）
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China（电子科技大学智能协同计算实验室）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；Peng Cheng Laboratory（鹏城实验室）
Unsupervised Multimodal 3D Medical Image Registration with Multilevel Correlation Balanced Optimization
College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）；Cornell University（康奈尔大学）
AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence（广东省计算人工智能模型与认知智能重点实验室）；School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；Pazhou Lab（琶洲实验室）；Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human（教育部健康智能感知与并行数字人工程研究中心）
Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification
Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）；College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；Key Laboratory of Collaborative Intelligent Systems of Ministry of Education, Xidian University（西安电子科技大学教育部协同智能系统重点实验室）；School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Academy of Artificial Intelligence, Inner Mongolia Normal University（内蒙古师范大学人工智能学院）
GeoBalance: Geometry-Aware Monitoring and Reconstruction with Asymmetric Optimization for Balanced Multimodal Learning
Beihang University（北京航空航天大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Amap, Alibaba Group（阿里巴巴集团高德）；Kuaishou Technology（快手科技）；Beijing Jiaotong University（北京交通大学）；Renmin University of China（中国人民大学）
Label What Matters: Modality-Balanced and Difficulty-Aware Multimodal Active Learning
Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education（北京交通大学计算机与信息技术学院；轨道交通控制与安全国家重点实验室；交通大数据与人工智能教育部重点实验室）；State Key Laboratory of Advanced Rail Autonomous Operation（格拉斯哥大学心理与神经科学学院）；School of Computer and Information Technology, Beijing Jiaotong University；School of Psychology & Neuroscience, University of Glasgow
MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation
School of Computing and Informatics, University of Louisiana at Lafayette（路易斯安那州立大学拉法叶分校计算机与信息学学院）；Louisiana Center for Health Innovation and College of Nursing & Health Sciences, University of Louisiana at Lafayette（路易斯安那州立大学拉法叶分校健康创新中心及护理与健康科学学院）；Massachusetts Eye and Ear, Harvard Medical School（哈佛医学院马萨诸塞眼耳医院）；Department of Computer Science, University of Central Florida（佛罗里达州立大学计算机科学系）；Department of Electrical Engineering and Computer Science, Florida Atlantic University（佛罗里达Atlantic大学电子工程与计算机科学系）
State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算北京先进创新中心）；Beihang University（北航）；Fudan University（复旦大学）；Peking University（北京大学）
Integrating Video and Text: A Balanced Approach to Multimodal Summary Generation and Evaluation
IRIT, University of Toulouse, France（IRIT，图卢兹大学，法国）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息通信研究所（I2R），A*STAR，新加坡）；CNRS, IRIT, France（CNRS，IRIT，法国）
Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究所）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation
College of Land Science and Technology, China Agricultural University（中国农业大学土地科学与技术学院）；Key Laboratory of Remote Sensing for Agri-Hazards, Ministry of Agriculture and Rural Affairs（农业农村部农业灾害遥感重点实验室）；Faculty of Geosciences and Engineering, Southwest Jiaotong University（西南交通大学地质科学与工程学院）；School of Artificial Intelligence, Sun Yat-Sen University（中山大学人工智能学院）；Henan Polytechnic University（河南理工大学）；Key Laboratory of Spatio-Temporal Information and Ecological Restoration of Mines, Ministry of Natural Resources of the People’s Republic of China（矿产资源时空信息与生态修复重点实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；National Supercomputing Center in Shenzhen, Shenzhen, China（深圳国家超算中心）；Ministry of Education Key Laboratory for Earth System Modeling and the Department of Earth System Science, Tsinghua University（地球系统模拟教育部重点实验室和清华大学地球系统科学系）
Modality-Balanced Learning for Multimedia Recommendation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
The University of Queensland（昆士兰大学）；Australian Institute for Machine Learning, Adelaide University（阿德莱德大学澳大利亚机器学习研究所）；University of Technology Sydney（悉尼科技大学）；Follow Me AI Pty LTD（Follow Me AI有限公司）
Courant Institute of Mathematical Sciences（库朗数学科学研究所）；New York University（纽约大学）；Vilcek Institute of Graduate Biomedical Sciences（维尔切克生物医学研究生科学研究所）
Multi-modal magnetic resonance imaging (MRI) is a crucial method for analyzing the human brain. It is usually used for diagnosing diseases and for making valuable decisions regarding the treatments - for instance, checking for gliomas in the human brain. With varying degrees of severity and detection, properly diagnosing gliomas is one of the most daunting and significant analysis tasks in modern-day medicine. Our primary focus is on working with different approaches to perform the segmentation of brain tumors in multimodal MRI scans. Now, the quantity, variability of the data used for trainin…
Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI
department of Radiology and Nuclear Medicine, Xuanwu Hospital, Capital Medical University（放射医学与核医学科，宣武医院，首都医科大学）；Beijing Key Laboratory of Magnetic Resonance Imaging and Brain Informatics（北京磁共振成像与脑信息学重点实验室）；Beijing United Imaging Research Institute of Intelligent Imaging（北京智能成像联合研究院）
Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Science and Technology of China（中国科学技术大学）；Arizona State University（亚利桑那州立大学）；Honda Research Institute, USA（本田美国研究所）
School of Computer Science and Engineering, Southeast University, Nanjing, China（计算机科学与工程学院，东南大学，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其跨学科应用关键实验室（东南大学），教育部，中国）
针对多模态视频模型中强模态主导导致的过拟合与模态不平衡问题，提出 MM 及改进的 B-MM 方法，通过动态调整模态混合比例提升泛化能力与鲁棒性。
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Contrast-enhanced T1 (T1ce) is one of the most essential magnetic resonance imaging (MRI) modalities for diagnosing and analyzing brain tumors, especially gliomas. In clinical practice, common MRI modalities such as T1, T2, and fluid attenuation inversion recovery are relatively easy to access while T1ce is more challenging considering the additional cost and potential risk of allergies to the contrast agent. Therefore, it is of great clinical necessity to develop a method to synthesize T1ce from other common modalities. Current paired image translation methods typically have the issue of requ…
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology（华南理工大学）；JD Explore Academy（京东探索研究院）；Nanyang Technological University（南洋理工大学）；Shanghai Jiao Tong University（上海交通大学）
