Large-scale pretrained models have proven immensely valuable in handling data-intensive modalities like text and image. However, fine-tuning these models for certain specialized modalities, such as protein sequence and cosmic ray, poses challenges due to the significant modality discrepancy and scarcity of labeled data. In this paper, we propose an end-to-end method, PaRe, to enhance cross-modal fine-tuning, aiming to transfer a large-scale pretrained model to various target modalities. PaRe employs a gating mechanism to select key patches from both source and target data. Through a modality-agnostic Patch Replacement scheme, these patches are preserved and combined to construct data-rich intermediate modalities ranging from easy to hard. By gradually intermediate modality generation, we can not only effectively bridge the modality gap to enhance stability and transferability of cross-modal fine-tuning, but also address the challenge of limited data in the target modality by leveraging enriched intermediate modality data. Compared with hand-designed, general-purpose, task-specific, and state-of-the-art cross-modal fine-tuning approaches, PaRe demonstrates superior performance across three challenging benchmarks, encompassing more than ten modalities.
Modality Unified Attack for Omni-Modality Person Re-Identification
College of Electrical and Information Engineering at Hunan University（湖南大学电气与信息工程学院）；National Engineering Research Center of Robot Visual Perception and Control Technology（机器人视觉感知与控制技术国家工程研究中心）；College of Information Science and Engineering at Hunan Normal University（湖南师范大学信息科学与工程学院）；Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing（湖南省智能计算与语言信息处理重点实验室）
Modality Bank: Learn multi-modality images across data centers without sharing medical data
Rutgers University（罗格斯大学）；SenseBrain Research（SenseBrain研究院）；National Heart Centre Singapore（新加坡国家心脏中心）；Duke-National University Of Singapore（杜克-新加坡国立大学）
Extended Cross-Modality United Learning for Unsupervised Visible-Infrared Person Re-identification
School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息工程学院）；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）
Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
Department of Computer Science, University of Kaiserslautern-Landau (RPTU)（科斯拉尔特伦大学计算机科学系）；SDS, German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；INRAE, UMR TETIS, University of Montpellier（蒙彼利埃大学UMR TETIS）；CIRAD, UMR TETIS, University of Montpellier（蒙彼利埃大学UMR TETIS）；INRIA, EVERGREEN, University of Montpellier（蒙彼利埃大学）
Diffusion based Zero-shot Medical Image-to-Image Translation for Cross Modality Segmentation
Athinoula A. Martinos Center for Biomedical Imaging（阿西诺拉·A.马蒂诺斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；Harvard University（哈佛大学）；Inria centre at Université Côte d’Azur（蔚蓝海岸大学国家信息与自动化研究所中心）；Temple University（天普大学）；Georg-August-University of Göttingen（哥廷根大学）
Cross-Modality Gait Recognition: Bridging LiDAR and Camera Modalities for Human Identification
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；University of Cordoba（科尔多瓦大学）；The Hong Kong Polytechnic University（香港理工大学）
Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection
The Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
University of Electronic Science and Technology of China（电子科技大学）；Intelligent Digital Media Technology Key Laboratory of Sichuan Province（四川省智能数字媒体技术重点实验室）
Mix-Modality Person Re-Identification: A New and Practical Paradigm
School of Computer Science and Technology, Wuhan University of Science and Technology（武汉科技大学计算机科学与技术学院）；National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（武汉大学多媒体软件国家工程研究中心、人工智能研究院、计算机学院）
Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Humanities and Social Sciences, Beihang University（北京航空航天大学人文与社会科学高等研究院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Informatics, Xiamen University（厦门大学信息学院）
Mix-modal Federated Learning for MRI Image Segmentation
School of Artificial Intelligence（人工智能学院）；School of Computer Science and Technology（计算机科学与技术学院）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
National University of Singapore（新加坡国立大学）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；National Neuroscience Institute（国家神经科学研究所）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）
Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation
College of Computer Science, Sichuan University（四川大学计算机学院）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；College of Electrical Engineering, Sichuan University（四川大学电气工程学院）；School of Cyber Science and Engineering, Sichuan University（四川大学网络空间安全学院）
