Cost-effective depth and infrared sensors as alternatives to usual RGB sensors are now a reality, and have some advantages over RGB in domains like autonomous navigation and remote sensing. As such, building computer vision and deep learning systems for depth and infrared data are crucial. However, large labeled datasets for these modalities are still lacking. In such cases, transferring knowledge from a neural network trained on a well-labeled large dataset in the source modality (RGB) to a neural network that works on a target modality (depth, infrared, etc.) is of great value. For reasons like memory and privacy, it may not be possible to access the source data, and knowledge transfer needs to work with only the source models. We describe an effective solution, SOCKET: SOurce-free Cross-modal KnowledgE Transfer for this challenging task of transferring knowledge from one source modality to a different target modality without access to task-relevant source data. The framework reduces the modality gap using paired task-irrelevant data, as well as by matching the mean and variance of the target features with the batch-norm statistics that are present in the source models. We show through extensive experiments that our method significantly outperforms existing source-free methods for classification tasks which do not account for the modality gap.
Cross-modality Person re-identification with Shared-Specific Feature Transfer
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Key Laboratory of Electromagnetic Space Information, Chinese Academy of Science（中国科学院电磁空间信息重点实验室）；Damo Academy, Alibaba Group（阿里巴巴达摩院）；Baidu Research(USA)（百度研究院（美国））
跨模态人物再识别（Cross-modality person re-identification，cm-ReID）是智能视频分析的一项具有挑战性的关键技术。现有的工作主要集中在学习共同的表示，通过嵌入不同的模态到同一个特征空间。然而，只学习共同特征意味着大量的信息损失，降低了特征区分度的上界。在本文中，我们解决了上述限制，提出了一种新的跨模态共享特定的特征转移算法（称为cm-SSFT），探索模态共享信息和模态特定的特征，以提高重新识别性能的潜力。我们根据共享特征对不同模态样本的相似度进行建模，然后在模态之间传递共享特征和特定特征。我们还提出了一种互补的特征学习策略，包括模态适应，项目对抗学习和重建增强，分别学习每种模态的区分性和互补性共享和特定特征。整个cm-SSFT算法可以以端到端的方式进行训练。我们进行了全面的实验，以验证整体算法的优越性和每个组件的有效性。该算法在两个主流基准数据集SYSU-MM 01和RegDB上的性能分别比最先进的算法高出22.5%和19.3%mAP。
SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人中心）；Princeton University（普林斯顿大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
Breaking the Modality Wall: Time-step Mixup for Efficient Spiking Knowledge Transfer from Static to Event Domain
Ningbo University（宁波大学）；Nanyang Technological University（南洋理工大学）；Merchants’ Guild Economics and Cultural（商帮经济与文化）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Robust Brain Tumor Segmentation with Incomplete MRI Modalities Using Hölder Divergence and Mutual Information-Enhanced Knowledge Transfer
MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University（深圳MSU-BIT大学应用数学联合研究中心）；Institute of Control Theory and Control Engineering, School of Automation, Beijing Institute of Technology（北京理工大学控制理论与控制工程研究所）；School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））
CSDN: Cross-modal Shape-transfer Dual-refinement Network for Point Cloud Completion
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Delft University of Technology（代尔夫特理工大学）；Lingnan University（岭南大学）；The Hong Kong Polytechnic University（香港理工大学）
X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；The Future Network of Intelligence Institute（未来网络智能研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shanghai Jiao Tong University（上海交通大学）；Sun Yat-sen University（中山大学）
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Georgia Institute of Technology（佐治亚理工学院）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）；School of Interactive Computing, Georgia Institute of Technology（佐治亚理工学院交互计算学院）；Emory University（埃默里大学）
MIST GAN: Modality Imputation Using Style Transfer for MRI
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；Healthcare Technology Innovation Centre（医疗技术创新中心）；Sri Rama Chandra Institute of Higher Education and Research（斯里·拉马·钱德拉高等教育与研究所）
Unlocking Optical Prior: Spectrum-Guided Knowledge Transfer for SAR Generalized Category Discovery
College of Electronic Science and Technology, National University of Defense Technology（电子科学与技术学院，国防科技大学）；State Key Laboratory of Complex System Simulation and Modeling Technology（复杂系统仿真与建模技术国家重点实验室）
DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial People’s Hospital（广东省人民医院）；Guangdong Academy of Medical Sciences（广东省医学科学院）
XTransfer: Modality-Agnostic Few-Shot Model Transfer for Human Sensing at the Edge
Macquarie University, Sydney, NSW, Australia（麦考瑞大学，悉尼，新南威尔士州，澳大利亚）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）；The University of Auckland, Auckland, New Zealand（奥克兰大学，奥克兰，新西兰）
Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science（计算机学院）；Wuhan University（武汉大学）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（多媒体与网络通信工程湖北省重点实验室）；Centre for Frontier AI Research（前沿人工智能研究中心）；Agency for Science, Technology and Research（新加坡科技研究局）；University of Maryland, College Park（马里兰大学帕克分校）；University of Toronto（多伦多大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Deep Doubly Supervised Transfer Network for Diagnosis of Breast Cancer with Imbalanced Ultrasound Imaging Modalities
Shanghai Institute for Advanced Communication and Data Science（上海先进通信与数据科学研究院）；The First Affiliated Hospital of Anhui Medical University（安徽医科大学第一附属医院）；Fudan University Shanghai Cancer Center（复旦大学附属肿瘤医院）；Shanghai Medical College, Fudan University（复旦大学上海医学院）；School of Science, Shanghai University（上海大学理学院）
Transferable Low-Rank Convolutional Bases for Onboarding Unseen Medical Imaging Modalities
Bangladesh University of Engineering and Technology (BUET)（孟加拉国工程技术大学）；Osaka Metropolitan University（大阪都市大学）；Chittagong University of Engineering and Technology (CUET)（吉大港工程技术大学）；Kochi University of Technology（高知工科大学）
