Multi-modal 3D object detection is pivotal for autonomous driving, integrating complementary sensors like LiDAR and cameras. However, its real-world reliability is challenged by transient data interruptions and missing, where modalities can momentarily drop due to hardware glitches, adverse weather, or occlusions. This poses a critical risk, especially during a simultaneous modality drop, where the vehicle is momentarily blind. To address this problem, we introduce ModalPatch, the first plug-and-play module designed to enable robust detection under arbitrary modality-drop scenarios. Without requiring architectural changes or retraining, ModalPatch can be seamlessly integrated into diverse detection frameworks. Technically, ModalPatch leverages the temporal nature of sensor data for perceptual continuity, using a history-based module to predict and compensate for transiently unavailable features. To improve the fidelity of the predicted features, we further introduce an uncertainty-guided cross-modality fusion strategy that dynamically estimates the reliability of compensated features, suppressing biased signals while reinforcing informative ones. Extensive experiments show that ModalPatch consistently enhances both robustness and accuracy of state-of-the-art 3D object detectors under diverse modality-drop conditions.
RXFOOD: Plug-in RGB-X Fusion for Object of Interest Detection
Cleveland State University（克利夫兰州立大学）；Centre for Frontier AI Research (CFAR)（前沿人工智能研究中心）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Institute of High Performance Computing (IHPC)（高性能计算研究所）；Brookhaven National Laboratory（布鲁克海文国家实验室）
The Chinese University of Hong Kong（香港中文大学）；University of Michigan（密歇根大学）；Imsight Medical Technology Inc.（Imsight医疗科技公司）
Convolutional networks (ConvNets) have achieved great successes in various challenging vision tasks. However, the performance of ConvNets would degrade when encountering the domain shift. The domain adaptation is more significant while challenging in the field of biomedical image analysis, where cross-modality data have largely different distributions. Given that annotating the medical data is especially expensive, the supervised transfer learning approaches are not quite optimal. In this paper, we propose an unsupervised domain adaptation framework with adversarial learning for cross-modality…
Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation
College of Computer Science, Sichuan University（四川大学计算机学院）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；College of Electrical Engineering, Sichuan University（四川大学电气工程学院）；School of Cyber Science and Engineering, Sichuan University（四川大学网络空间安全学院）
SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Deep neural networks (DNNs) usually fail to generalize well to outside of distribution (OOD) data, especially in the extreme case of single domain generalization (single-DG) that transfers DNNs from single domain to multiple unseen domains. Existing single-DG techniques commonly devise various data-augmentation algorithms, and remould the multi-source domain generalization methodology to learn domain-generalized (semantic) features. Nevertheless, these methods are typically modality-specific, thereby being only applicable to one single modality (e.g., image). In contrast, we target a versatile…
Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；Department of Computing, Imperial College London（帝国理工学院计算机系）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）；Imsight Medical Technology, Co, Ltd., China（中国医视医疗科技有限公司）；School of Data Science, Fudan University, China（复旦大学数据科学学院）
Deep convolutional networks have demonstrated the state-of-the-art performance on various medical image computing tasks. Leveraging images from different modalities for the same analysis task holds clinical benefits. However, the generalization capability of deep models on test data with different distributions remain as a major challenge. In this paper, we propose the PnPAdaNet (plug-and-play adversarial domain adaptation network) for adapting segmentation networks between different modalities of medical images, e.g., MRI and CT. We propose to tackle the significant domain shift by aligning t…
Unified-modal Salient Object Detection via Adaptive Prompt Learning
Anhui University（安徽大学）；School of Computer Science and Technology（计算机科学与技术学院）；School of Artificial Intelligence（人工智能学院）；Institute of Physical Science and Information Technology（物理科学与技术学院）
3D object detection with multi-sensors is essential for an accurate and reliable perception system of autonomous driving and robotics. Existing 3D detectors significantly improve the accuracy by adopting a two-stage paradigm which merely relies on LiDAR point clouds for 3D proposal refinement. Though impressive, the sparsity of point clouds, especially for the points far away, making it difficult for the LiDAR-only refinement module to accurately recognize and locate objects.To address this problem, we propose a novel multi-modality two-stage approach named FusionRCNN, which effectively and ef…
Gesture recognition is getting more and more popular due to various application possibilities in human-machine interaction. Existing multi-modal gesture recognition systems take multi-modal data as input to improve accuracy, but such methods require more modality sensors, which will greatly limit their application scenarios. Therefore we propose an end-to-end multi-task learning framework in training 2D convolutional neural networks. The framework can use the depth modality to improve accuracy during training and save costs by using only RGB modality during inference. Our framework is trained …
HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；School of Mathematics, Southeast University（东南大学数学学院）
MGML: A Plug-and-Play Meta-Guided Multi-Modal Learning Framework for Incomplete Multimodal Brain Tumor Segmentation
Nanchang University（南昌大学）；Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University（中山大学孙逸仙纪念医院）；Zhujiang Hospital, Southern Medical University（南方医科大学珠江医院）；University of California Los Angeles（加州大学洛杉矶分校）；Hubei University of Economics（湖北经济学院）；Xi'an Jiaotong Liverpool University（西交利物浦大学）；Southeast University（东南大学）
Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captio…
National University of Singapore（新加坡国立大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Shandong University（山东大学）；Shandong Artificial Intelligence Institute（山东人工智能研究院）；University of Florence（佛罗伦萨大学）
Making each modality in multi-modal data contribute is of vital importance to learning a versatile multi-modal model. Existing methods, however, are often dominated by one or few of modalities during model training, resulting in sub-optimal performance. In this paper, we refer to this problem as modality bias and attempt to study it in the context of multi-modal classification systematically and comprehensively. After stepping into several empirical analysis, we recognize that one modality affects the model prediction more just because this modality has a spurious correlation with instance lab…
Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Humanities and Social Sciences, Beihang University（北京航空航天大学人文与社会科学高等研究院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Informatics, Xiamen University（厦门大学信息学院）
Northwestern Polytechnical University（西北工业大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
Temporal action localization aims at localizing action instances from untrimmed videos. Existing works have designed various effective modules to precisely localize action instances based on appearance and motion features. However, by treating these two kinds of features with equal importance, previous works cannot take full advantage of each modality feature, making the learned model still sub-optimal. To tackle this issue, we make an early effort to study temporal action localization from the perspective of multi-modality feature learning, based on the observation that different actions exhi…
Rutgers University（罗格斯大学）；SenseBrain Research（SenseBrain研究院）；National Heart Centre Singapore（新加坡国家心脏中心）；Duke-National University Of Singapore（杜克-新加坡国立大学）
Multi-modality images have been widely used and provide comprehensive information for medical image analysis. However, acquiring all modalities among all institutes is costly and often impossible in clinical settings. To leverage more comprehensive multi-modality information, we propose a privacy secured decentralized multi-modality adaptive learning architecture named ModalityBank. Our method could learn a set of effective domain-specific modulation parameters plugged into a common domain-agnostic network. We demonstrate by switching different sets of configurations, the generator could outpu…
In many visual systems, visual tracking often bases on RGB image sequences, in which some targets are invalid in low-light conditions, and tracking performance is thus affected significantly. Introducing other modalities such as depth and infrared data is an effective way to handle imaging limitations of individual sources, but multi-modal imaging platforms usually require elaborate designs and cannot be applied in many real-world applications at present. Near-infrared (NIR) imaging becomes an essential part of many surveillance cameras, whose imaging is switchable between RGB and NIR based on…
Geometric Intelligence（几何智能）；Uber AI Labs（优步人工智能实验室）；University of Wyoming（怀俄明大学）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）；University of Freiburg（弗莱堡大学）
Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting way to synthesize novel images by performing gradient ascent in the latent space of a generator network to maximize the activations of one or multiple neurons in a separate classifier network. In this paper we extend this method by introducing an additional prior on the latent code, improving both sample quality and sample diversity, leading to a state-of-the-art generative model that produces high quality images at higher resolutions (227x…
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modal…
Cardiac magnetic resonance imaging (CMR) is a noninvasive imaging modality that provides a comprehensive evaluation of the cardiovascular system. The clinical utility of CMR is hampered by long acquisition times, however. In this work, we propose and validate a plug-and-play (PnP) method for CMR reconstruction from undersampled multi-coil data. To fully exploit the rich image structure inherent in CMR, we pair the PnP framework with a deep learning (DL)-based denoiser that is trained using spatiotemporal patches from high-quality, breath-held cardiac cine images. The resulting "PnP-DL" method …
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Meta AI
In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments. However, this process is expensive; massive amounts of interactions are needed for the model to generalize well. Moreover, this process is repeated whenever there is a change in the task type or the goal modality. We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target…
MPISuperRes-PnP: A Super-Resolution Zero-Shot Plug-and-Play Reconstruction Algorithm for Magnetic Particle Imaging
Institute for the Protection of Terrestrial Infrastructures, German Aerospace Center (DLR)（德国航空航天中心陆地基础设施保护研究所）；Hochschule Darmstadt（达姆施塔特应用科学大学）；Data Science Institute, European University of Technology（欧洲技术大学数据科学研究所）；Technische Hochschule Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用技术大学）
Can BEV Perception Gracefully Degrade under Sensor Failures?
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System（天津智能无人群技术与系统重点实验室）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理重点实验室，中华人民共和国教育部）
Great Bay University（大湾区大学）；Nanyang Technological University（南洋理工大学）；University of Oulu（奥卢大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University（深圳大学）
Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs
School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）；Nanyang Technological University, Singapore（新加坡南洋理工大学）；University College London（伦敦大学学院）；Guangzhou University（广州大学）；Wuhan University（武汉大学）；Nanjing University（南京大学）
X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation
Shanghai Engineering Research Center of AI & Robotics（上海人工智能与机器人工程研究中心）；Academy for Engineering & Technology, Fudan University（复旦大学工程技术研究院）；Shanghai Key Lab of Intelligent Information Processing（上海市智能信息处理重点实验室）；School of Computer Science, Fudan University（复旦大学计算机学院）
MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection
Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences)（计算机科学与技术学院，齐鲁工业大学（山东科学院））；School of Computing, National University of Singapore（computing 学院，新加坡国立大学）；School of Computing and Information Technology, Great Bay University（computing 与信息学院，大湾大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）
CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion
Wuhan University of Technology（武汉理工大学）；University of North Carolina at Charlotte（北卡罗来纳州立大学）；Wuhan University（武汉大学）；Southwest Jiaotong University（西南交通大学）；The Hong Kong Polytechnic University（香港理工大学）
