The goal of multi-modal learning is to use complimentary information on the relevant task provided by the multiple modalities to achieve reliable and robust performance. Recently, deep learning has led significant improvement in multi-modal learning by allowing for the information fusion in the intermediate feature levels. This paper addresses a problem of designing robust deep multi-modal learning architecture in the presence of imperfect modalities. We introduce deep fusion architecture for object detection which processes each modality using the separate convolutional neural network (CNN) and constructs the joint feature map by combining the intermediate features from the CNNs. In order to facilitate the robustness to the degraded modalities, we employ the gated information fusion (GIF) network which weights the contribution from each modality according to the input feature maps to be fused. The weights are determined through the convolutional layers followed by a sigmoid function and trained along with the information fusion network in an end-to-end fashion. Our experiments show that the proposed GIF network offers the additional architectural flexibility to achieve robust performance in handling some degraded modalities, and show a significant performance improvement based on Single Shot Detector (SSD) for KITTI dataset using the proposed fusion network and data augmentation schemes.
CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；School of Computer Science, the University of Sydney（悉尼大学计算机学院）；China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司）
Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion
The Chinese University of Hong Kong（香港中文大学）；Imperial College Longdon（伦敦帝国理工学院）；Imsight Medical Technology Co. Ltd.（艾迈医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）
RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation
Faculty of Engineering and Information Technology, University of Technology Sydney（工程与信息技术学院，悉尼技术大学）；PCA Lab, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology（教育部高维信息智能感知与系统重点实验室，南京理工大学计算机科学与工程学院）；Bionic Vision Systems Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所生物视觉系统实验室）；Research Center for Industries of the Future and the School of Engineering, Westlake University（未来产业研究中心和工程学院，西湖大学）；OPPO Research, Seattle, WA 98101 USA（OPPO研究，美国华盛顿州西雅图98101）
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Central China Normal University（华中师范大学）；Huazhong University of Science and Technology（华中科技大学）；Guangzhou University of Chinese Medicine（广州中医药大学）；The First Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第一附属医院）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimation
Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）
3D Gated Recurrent Fusion for Semantic Scene Completion
The University of Adelaide（阿德莱德大学）；Nanjing University of Science and Technology（南京理工大学）；Northwestern Polytechnical University（西北工业大学）；ETH Zurich（苏黎世联邦理工学院）
Mask-adaptive Gated Convolution and Bi-directional Progressive Fusion Network for Depth Completion
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Foshan Graduate School of Innovation, Northeastern University（东北大学佛山创新研究生院）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University（东北大学工业智能与系统优化国家前沿科学中心）
DeepDualMapper: A Gated Fusion Network for Automatic Map Extraction using Aerial Images and Trajectories
School of Computer Science, Fudan University（复旦大学计算机学院）；Systems and Shanghai Key Laboratory of Data Science, Fudan University（复旦大学系统与上海数据科学重点实验室）；Shanghai Insitute of Intelligent Electroics & Systems（上海智能电子与系统研究所）；Bytedance AI Lab（字节跳动AI实验室）；Singapore Management University（新加坡管理大学）
Fuzhou University（福州大学）；Tencent AI Lab（腾讯人工智能实验室）；Jiangnan University（江南大学）；South China University of Technology（华南理工大学）；City University of Hong Kong（香港城市大学）
MoCTEFuse: Illumination-Gated Mixture of Chiral Transformer Experts for Multi-Level Infrared and Visible Image Fusion
School of Optics and Photonics, Beijing Institute of Technology（光学与光电学院，北京理工大学）；School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）
Dual-path CNN with Max Gated block for Text-Based Person Re-identification
Nanjing University of information science and Technology（南京信息工程大学）；Xinjiang University（新疆大学）；Nanjing Institute of Technology（南京工程学院）；KingSaud University（沙特国王大学）
State Key Laboratory of Information Security (SKLOIS), IIE, CAS（中国科学院信息工程研究所信息安全国家重点实验室）；Tencent AI Lab（腾讯人工智能实验室）；City University of Hong Kong（香港城市大学）；Nanjing University of Science and Technology（南京理工大学）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Electrical Engineering and Computer Science, University of California, Merced（加州大学默塞德分校电气与计算机科学学院）
WING: A Window-Prior-Based Generative Network with Gated Inception for Cross-Modality CT Synthesis
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg（模式识别实验室，埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Department of Orthodontics and Orofacial Orthopaedics, Friedrich-Alexander-Universität Erlangen-Nürnberg（正畸与口腔颌面正畸科，埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Siemens Healthineers（西门子医疗）；Institute of Medical Technology, Peking University（北京大学医学技术研究所）
Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding
School of Computer Science and Artificial Intelligence, Changzhou University（常州大学计算机与人工智能学院）；Department of Computer Science, University of Sheffield（谢菲尔德大学计算机系）
