SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality
Center for Artificial Intelligence and Robotics, HKISI, CAS（中国科学院香港创新研究院人工智能与机器人中心）；Princeton University（普林斯顿大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
Chenyang Lei, Liyi Chen, Jun Cen, Xiao Chen, Zhen Lei, Felix Heide, Ziwei Liu, Qifeng Chen, Zhaoxiang Zhang
Foundation models like ChatGPT and Sora that are trained on a huge scale of data have made a revolutionary social impact. However, it is extremely challenging for sensors in many different fields to collect similar scales of natural images to train strong foundation models. To this end, this work presents a simple and effective framework SimMAT to study an open problem: the transferability from vision foundation models trained on natural RGB images to other image modalities of different physical properties (e.g., polarization). SimMAT consists of a modality-agnostic transfer layer (MAT) and a pretrained foundation model. We apply SimMAT to a representative vision foundation model Segment Anything Model (SAM) to support any evaluated new image modality. Given the absence of relevant benchmarks, we construct a new benchmark to evaluate the transfer learning performance. Our experiments confirm the intriguing potential of transferring vision foundation models in enhancing other sensors' performance. Specifically, SimMAT can improve the segmentation performance (mIoU) from 22.15% to 53.88% on average for evaluated modalities and consistently outperforms other baselines. We hope that SimMAT can raise awareness of cross-modal transfer learning and benefit various fields for better results with vision foundation models.
Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；University of Oxford（牛津大学）；Drexel University（德雷塞尔大学）；Beijing University of Technology（北京理工大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Xi’an Jiaotong University（西安交通大学）
SenseTime Research（商汤科技研究院）；University of Sydney（悉尼大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De-CLIP can learn generic visual features more efficiently. Ins…
Although many studies have successfully applied transfer learning to medical image segmentation, very few of them have investigated the selection strategy when multiple source tasks are available for transfer. In this paper, we propose a prior knowledge guided and transferability based framework to select the best source tasks among a collection of brain image segmentation tasks, to improve the transfer learning performance on the given target task. The framework consists of modality analysis, RoI (region of interest) analysis, and transferability estimation, such that the source task selectio…
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Monash University（莫纳什大学）；Temple University（天普大学）；Peng Cheng Laboratory（鹏城实验室）
SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）
China University of Geosciences（中国地质大学）；Nanjing University of Science and Technology（南京理工大学）；University of Chinese Academy of Sciences（中国科学院大学）
RGB-Infrared person re-identification (RGB-IR ReID) aims to associate people across disjoint RGB and IR camera views. Currently, state-of-the-art performance of RGB-IR ReID is not as impressive as that of conventional ReID. Much of that is due to the notorious modality bias training issue brought by the single-modality ImageNet pre-training, which might yield RGB-biased representations that severely hinder the cross-modality image retrieval. This paper makes first attempt to tackle the task from a pre-training perspective. We propose a self-supervised pre-training solution, named Modality-Awar…
Optimizing Multi-Modality Trackers via Significance-Regularized Tuning
School of Artificial Intelligence, Xidian University（电子科技大学人工智能学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）
Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Academy for Engineering & Technology, Fudan University（复旦大学工程技术学院）；Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理重点实验室）；Shanghai Engineering Research Center of AI & Robotics（上海人工智能与机器人工程研究中心）；Engineering Research Center of AI & Robotics, Ministry of Education（教育部人工智能与机器人工程研究中心）
Recent studies have shown that adversarial examples hand-crafted on one white-box model can be used to attack other black-box models. Such cross-model transferability makes it feasible to perform black-box attacks, which has raised security concerns for real-world DNNs applications. Nevertheless, existing works mostly focus on investigating the adversarial transferability across different deep models that share the same modality of input data. The cross-modal transferability of adversarial perturbation has never been explored. This paper investigates the transferability of adversarial perturba…
XTransfer: Modality-Agnostic Few-Shot Model Transfer for Human Sensing at the Edge
Macquarie University, Sydney, NSW, Australia（麦考瑞大学，悉尼，新南威尔士州，澳大利亚）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）；The University of Auckland, Auckland, New Zealand（奥克兰大学，奥克兰，新西兰）
FPL+: Filtered Pseudo Label-based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation
School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China（电子科技大学机械与电气工程学院）；Shanghai AI laboratory（上海人工智能实验室）；West China Hospital, Sichuan University（四川大学华西医院）；West China Biomedical Big Data Center, West China Hospital, Sichuan University（四川大学华西医院华西生物医学大数据中心）；SenseTime Research（商汤科技研究院）
Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models
School of Electrical Engineering and Computer Science, the University of Queensland（电气工程与计算机科学学院，昆士兰大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）
USF-MAE: Ultrasound Self-Supervised Foundation Model with Masked Autoencoding
Department of Systems and Computer Engineering, Carleton University（系统与计算机工程系，卡尔顿大学）；Department of Clinical Science and Translational Medicine, University of Ottawa（临床科学与转化医学系，渥太华大学）；Department of Obstetrics and Gynecology, University of Ottawa（妇产科系，渥太华大学）；School of Epidemiology and Public Health, University of Ottawa（流行病学与公共卫生学院，渥太华大学）；Department of Methodological and Implementation Research, Ottawa Hospital Research Institute（方法学与实施研究系，渥太华医院研究学院）；Department of Acute Care Research, Ottawa Hospital Research Institute（急症护理研究系，渥太华医院研究学院）
Fundus to Fluorescein Angiography Video Generation as a Retinal Generative Foundation Model
The Hong Kong Polytechnic University（香港理工大学）；Swiss Federal Institute of Technology Lausanne (EPFL)（瑞士洛桑联邦理工学院）；Clemson University（克莱姆森大学）；University of Science and Technology of China（中国科学技术大学）；Research Centre for SHARP Vision (RSCV), The Hong Kong Polytechnic University（香港理工大学SHARP视觉研究中心）；Centre for Eye and Vision Research (CEVR)（眼科视觉研究中心）
SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition
Hong Kong University of Science and Technology（香港科技大学）；Tongji University（同济大学）；Tongji Hospital（同济医院）；Tongji Institute of Digestive Disease（同济消化病研究所）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Google Research（谷歌研究院）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
Vision-language (VL) pre-training has recently gained much attention for its transferability and flexibility in novel concepts (e.g., cross-modality transfer) across various visual tasks. However, VL-driven segmentation has been under-explored, and the existing approaches still have the burden of acquiring additional training images or even segmentation annotations to adapt a VL model to downstream segmentation tasks. In this paper, we introduce a novel image-free segmentation task where the goal is to perform semantic segmentation given only a set of the target semantic categories, but withou…
MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training
The Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen People’s Hospital（深圳市人民医院）；The University of Hong Kong（香港大学）；Chinese PLA General Hospital（中国人民解放军总医院）
