Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance.
The most challenging problems for guided DSR are how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhance block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.
Infrared and Visible Image Fusion with Hierarchical Human Perception
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications（重庆邮电大学图像认知重点实验室）
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
Xi’an Jiaotong University（西安交通大学）；ETH Zürich（苏黎世联邦理工学院）；Northwestern Polytechnical University（西北工业大学）；Research and Development Institute of Northwestern Polytechnical University in Shenzhen（西北工业大学深圳研究院）；Harvard University（哈佛大学）；University of Würzburg（维尔茨堡大学）
Modality-Aware Infrared and Visible Image Fusion with Target-Aware Supervision
Independent researcher（独立研究者）；Dept. of Computer Science Baylor University（计算机科学系 巴里尔大学）；Dept. of Computer Science Engineering University of Connecticut（计算机科学工程系 佛罗里达大学）；Dept. of Computer Science New York University（计算机科学系 新 york 大学）
Interactive Spatial-Frequency Fusion Mamba for Multi-Modal Image Fusion
School of Future Technology, Dalian University of Technology and the Key Laboratory of Data Science and Smart Education (Hainan Normal University), Ministry of Education（未来技术学院，大连理工大学和数据科学与智能教育关键实验室（海南师范大学），教育部）；Affiliated Zhongshan Hospital of Dalian University（大连大学附属中山医院）；School of Computer Science and Artificial Intelligence, Wuhan University of Technology（计算机科学与人工智能学院，武汉理工大学）；Central Hospital of Dalian University of Technology（大连理工大学中心医院）；School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）
Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding
School of Computer Science and Artificial Intelligence, Changzhou University（常州大学计算机与人工智能学院）；Department of Computer Science, University of Sheffield（谢菲尔德大学计算机系）
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Central China Normal University（华中师范大学）；Huazhong University of Science and Technology（华中科技大学）；Guangzhou University of Chinese Medicine（广州中医药大学）；The First Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第一附属医院）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；School of Computer Science, the University of Sydney（悉尼大学计算机学院）；China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司）
FlexiD-Fuse: Flexible number of inputs multi-modal medical image fusion based on diffusion model
School of Physics（物理学院）；Optoelectronic Engineering, Foshan University（光电工程学院，佛山大学）；Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology（粤港澳联合智能微纳光电技术实验室）；Guangdong Provincial Key Laboratory of Industrial Intelligent Inspection Technology（广东省工业智能检测技术重点实验室）；School of Information Engineering（信息工程学院）；Automation, Kunming University of Science（自动化系，昆明理工大学）
Multi-modal Sensor Fusion for Auto Driving Perception: A Survey
University of California Los Angeles（加州大学洛杉矶分校）；Shanghai AI Lab（上海人工智能实验室）；Beijing Institute of Technology（北京理工大学）；East China Normal University（华东师范大学）
Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification
Bytedance AI Lab（字节跳动人工智能实验室）
利用视觉帧和音频已被实验证明有效地提高大规模视频分类。以往的视频分类研究主要集中在提取视频帧间的视觉内容分析及其时间特征聚合。相比之下，多模态数据融合是通过简单的运算符，如平均和级联。受双线性池在视觉和语言融合中的成功启发，我们引入多模态分解双线性池（MFB）来融合视觉和音频表示。我们结合MFB与不同的视频级功能，并探讨其在视频分类的有效性。在具有挑战性的Youtube-8 M v2数据集上的实验结果表明，MFB在大规模视频分类中的性能明显优于简单的融合方法。
Dissecting RGB-D Learning for Improved Multi-modal Fusion
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及跨学科应用国家重点实验室（东南大学））；Tsinghua University（清华大学）；College of Computer Science, Beijing University of Technology（北京理工大学计算机学院）
Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Digital Medical Research Center, School of Basic Medical Science, Fudan University（复旦大学基础医学院数字医学研究中心）；University of South Carolina（南卡罗来纳大学）
Local-to-Global Cross-Modal Attention-Aware Fusion for HSI-X Semantic Segmentation
International Institute for Earth System Science, Nanjing University（南京大学地球系统科学国际学院）；The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Trento（特伦托大学）
Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University（多源信息挖掘与安全广西重点实验室，广西师范大学）
