Infrared and visible image fusion aim to integrate modality strengths for visually enhanced, informative images. Visible imaging in real-world scenarios is susceptible to dynamic environmental brightness fluctuations, leading to texture degradation. Existing fusion methods lack robustness against such brightness perturbations, significantly compromising the visual fidelity of the fused imagery. To address this challenge, we propose the Brightness Adaptive multimodal dynamic fusion framework (BA-Fusion), which achieves robust image fusion despite dynamic brightness fluctuations. Specifically, we introduce a Brightness Adaptive Gate (BAG) module, which is designed to dynamically select features from brightness-related channels for normalization, while preserving brightness-independent structural information within the source images. Furthermore, we propose a brightness consistency loss function to optimize the BAG module. The entire framework is tuned via alternating training strategies. Extensive experiments validate that our method surpasses state-of-the-art methods in preserving multi-modal image information and visual fidelity, while exhibiting remarkable robustness across varying brightness levels. Our code is available: https://github.com/SunYM2020/BA-Fusion.
Domain adaptation is an important task to enable learning when labels are scarce. While most works focus only on the image modality, there are many important multi-modal datasets. In order to leverage multi-modality for domain adaptation, we propose cross-modal learning, where we enforce consistency between the predictions of two modalities via mutual mimicking. We constrain our network to make correct predictions on labeled data and consistent predictions across modalities on unlabeled target-domain data. Experiments in unsupervised and semi-supervised domain adaptation settings prove the eff…
Sichuan University（四川大学）；University of Electronic Science and Technology of China（电子科技大学）；Dalian University of Technology（大连理工大学）；National University of Defense Technology（国防科技大学）
Domain adaptation is critical for success when confronting with the lack of annotations in a new domain. As the huge time consumption of labeling process on 3D point cloud, domain adaptation for 3D semantic segmentation is of great expectation. With the rise of multi-modal datasets, large amount of 2D images are accessible besides 3D point clouds. In light of this, we propose to further leverage 2D data for 3D domain adaptation by intra and inter domain cross modal learning. As for intra-domain cross modal learning, most existing works sample the dense 2D pixel-wise features into the same size…
Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that directly applying existing methods usually results in performance instability at test time because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two compl…
Robust Multi-Modal Face Anti-Spoofing with Domain Adaptation: Tackling Missing Modalities, Noisy Pseudo-Labels, and Model Degradation
College of Computer and Cyber Security, Fujian Normal University（计算机与网络安全部，福建师范大学）；Department of Computer Science, National Tsing Hua University（计算机科学系，国立清华大学）
Boston University（波士顿大学）；NEC Labs America（NEC美国实验室）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）
Learning transferable and domain adaptive feature representations from videos is important for video-relevant tasks such as action recognition. Existing video domain adaptation methods mainly rely on adversarial feature alignment, which has been derived from the RGB image space. However, video data is usually associated with multi-modal information, e.g., RGB and optical flow, and thus it remains a challenge to design a better method that considers the cross-modal inputs under the cross-domain adaptation setting. To this end, we propose a unified framework for video domain adaptation, which si…
Radboud University Medical Center（拉德堡德大学医学中心）；The Netherlands Cancer Institute（荷兰癌症研究所）；Nanjing University of Information Science and Technology（南京信息工程大学）
Domain adaptation has been widely adopted to transfer styles across multi-vendors and multi-centers, as well as to complement the missing modalities. In this challenge, we proposed an unsupervised domain adaptation framework for cross-modality vestibular schwannoma (VS) and cochlea segmentation and Koos grade prediction. We learn the shared representation from both ceT1 and hrT2 images and recover another modality from the latent representation, and we also utilize proxy tasks of VS segmentation and brain parcellation to restrict the consistency of image structures in domain adaptation. After …
Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation
Northwestern University（西北大学）；Futurewei Technologies（华为未来技术研究院）；University of Electronic Science and Technology of China（电子科技大学）；University of Rochester（罗切斯特大学）
UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations
Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation
The Chinese University of Hong Kong（香港中文大学）；Imsight Medical Technology Co., Ltd.（影迈医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）；SIAT（深圳先进技术研究院）
Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation
East China University of Science and Technology（华东理工大学）；vivo Mobile Communication Co., Ltd（维沃移动通信有限公司）；Peng Cheng Laboratory（鹏城实验室）；Singapore Management University（新加坡管理大学）
PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network with a Benchmark at Cross-modality Cardiac Segmentation
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；Department of Computing, Imperial College London（帝国理工学院计算机系）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）；Imsight Medical Technology, Co, Ltd., China（中国医视医疗科技有限公司）；School of Data Science, Fudan University, China（复旦大学数据科学学院）
The Chinese University of Hong Kong（香港中文大学）；University of Leicester（莱斯特大学）
Domain adaptation is crucial for transferring the knowledge from the source labeled CT dataset to the target unlabeled MR dataset in abdominal multi-organ segmentation. Meanwhile, it is highly desirable to avoid the high annotation cost related to the target dataset and protect the source dataset privacy. Therefore, we propose an effective source-free unsupervised domain adaptation method for cross-modality abdominal multi-organ segmentation without source dataset access. The proposed framework comprises two stages. In the first stage, the feature map statistics-guided model adaptation combine…
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程研究中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Continual Test-Time Adaptation (CTTA) generalizes conventional Test-Time Adaptation (TTA) by assuming that the target domain is dynamic over time rather than stationary. In this paper, we explore Multi-Modal Continual Test-Time Adaptation (MM-CTTA) as a new extension of CTTA for 3D semantic segmentation. The key to MM-CTTA is to adaptively attend to the reliable modality while avoiding catastrophic forgetting during continual domain shifts, which is out of the capability of previous TTA or CTTA methods. To fulfill this gap, we propose an MM-CTTA method called Continual Cross-Modal Adaptive Clu…
Digital Medical Research Center, School of Basic Medical Science, Fudan University（复旦大学基础医学院数字医学研究中心）；Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention（上海市医学影像计算与计算机辅助干预重点实验室）
Unsupervised domain adaptation (UDA) aims to learn a model trained on source domain and performs well on unlabeled target domain. In medical image segmentation field, most existing UDA methods depend on adversarial learning to address the domain gap between different image modalities, which is ineffective due to its complicated training process. In this paper, we propose a simple yet effective UDA method based on frequency and spatial domain transfer uner multi-teacher distillation framework. In the frequency domain, we first introduce non-subsampled contourlet transform for identifying domain…
Medical School of Nanjing University（南京大学医学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学国家健康医疗大数据研究院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computer Science and Technology, Nanjing University of Science and Technology（南京理工大学计算机科学与技术学院）；Department of Radiology, Nanjing Drum Tower Hospital, Nanjing University Medical School（南京大学医学院附属鼓楼医院放射科）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Department of Research and Development, Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司研发部）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）
Accurate segmentation of brain tumors from multi-modal Magnetic Resonance (MR) images is essential in brain tumor diagnosis and treatment. However, due to the existence of domain shifts among different modalities, the performance of networks decreases dramatically when training on one modality and performing on another, e.g., train on T1 image while performing on T2 image, which is often required in clinical applications. This also prohibits a network from being trained on labeled data and then transferred to unlabeled data from a different domain. To overcome this, unsupervised domain adaptat…
Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery
College of Computer Science, Nankai University（南开大学计算机学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
SDC-UDA: Volumetric Unsupervised Domain Adaptation Framework for Slice-Direction Continuous Cross-Modality Medical Image Segmentation
Yonsei University（延世大学）；Harvard Medical School（哈佛医学院）；Naver AI Lab（Naver人工智能实验室）；Naver Cloud（Naver云）；Probe Medical Inc.（Probe医疗公司）；Martinos Center for Biomedical Imaging（马丁诺斯生物医学成像中心）；Korea Institute of Science and Technology（韩国科学技术研究院）
Learning Progressive Adaptation for Multi-Modal Tracking
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（Surrey大学视觉、语音和信号处理中心）
Due to the lack of properly annotated medical data, exploring the generalization capability of the deep model is becoming a public concern. Zero-shot learning (ZSL) has emerged in recent years to equip the deep model with the ability to recognize unseen classes. However, existing studies mainly focus on natural images, which utilize linguistic models to extract auxiliary information for ZSL. It is impractical to apply the natural image ZSL solutions directly to medical images, since the medical terminology is very domain-specific, and it is not easy to acquire linguistic models for the medical…
CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Centre for Vision, Speech and Signal Processing (CVSSP)（视觉、语音和信号处理中心）；University of Surrey（塞维利亚大学）；Surrey Institute for People-Centred AI（以人为本的人工智能研究所）
Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）
Assuming the source label space subsumes the target one, Partial Video Domain Adaptation (PVDA) is a more general and practical scenario for cross-domain video classification problems. The key challenge of PVDA is to mitigate the negative transfer caused by the source-only outlier classes. To tackle this challenge, a crucial step is to aggregate target predictions to assign class weights by up-weighing target classes and down-weighing outlier classes. However, the incorrect predictions of class weights can mislead the network and lead to negative transfer. Previous works improve the class weig…
Friedrich-Alexander-University Erlangen-Nuremberg（弗里德里希-亚历山大-埃尔朗根-纽伦堡大学）；University of Leeds（利兹大学）；CISTIB, Centre for Computational Imaging and Simulation Technologies in Biomedicine（生物医学计算成像与模拟技术中心）；School of Computing（计算机学院）；LICAMM Leeds Institute of Cardiovascular and Metabolic Medicine（利兹心血管与代谢医学研究所）；School of Medicine（医学院）
Deep learning models are sensitive to domain shift phenomena. A model trained on images from one domain cannot generalise well when tested on images from a different domain, despite capturing similar anatomical structures. It is mainly because the data distribution between the two domains is different. Moreover, creating annotation for every new modality is a tedious and time-consuming task, which also suffers from high inter- and intra- observer variability. Unsupervised domain adaptation (UDA) methods intend to reduce the gap between source and target domains by leveraging source domain labe…
