Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. Moreover, the data used for pretraining foundation models are usually invisible and very different from the target data of downstream tasks. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task that has a quite different architecture with only downstream target data. Existing transfer learning or knowledge distillation methods depend on either the same model structure or finetuning of the foundation model. Thus, naively introducing these methods can be either infeasible or very inefficient. To address this, we propose a Task-Driven Model Reprogramming (TDMR) framework. Specifically, we reprogram the foundation model to project the knowledge into a proxy space, which alleviates the adverse effect of task mismatch and domain inconsistency. Then, we reprogram the target model via progressive distillation from the proxy space to efficiently learn the knowledge from the reprogrammed foundation model. TDMR is compatible with different pre-trained model types (CNN, transformer or their mix) and limited target data, and promotes the wide applications of vision foundation models to downstream tasks in a cost-effective manner. Extensive experiments on different downstream classification tasks and target model structures demonstrate the effectiveness of our methods with both CNNs and transformer foundation models.
Foundation Model-Driven Classification of Atypical Mitotic Figures with Domain-Aware Training Strategies
Wrocław University of Science and Technology(沃拉什大学科学与技术学院);Cancer Center Sp. z o. o.(癌症中心有限公司);Hospital for Lung Diseases - Rebirth(肺病医院-重生);Maria Sklodowska-Curie National Research Institute of Oncology(玛丽·斯克洛多夫斯卡国家癌症研究院);Pathology Department(病理部门);th Military Research Hospital in Bydgoszcz(比多斯茨第10军科研医院);Department of Tumor Pathology and Pathomorphology(肿瘤病理与组织病理学部门);Oncology Centre Prof. Franciszek Łukaszczyk Memorial Hospital(弗雷德里克·卢卡斯茨基纪念医院肿瘤中心);Doctoral School of Medical and Health Sciences, Nicolaus Copernicus University in Toruń(医学与健康科学博士学院,尼古拉斯·哥白尼大学图努恩分校);Department of Obstetrics, Gynaecology and Oncology, Collegium Medicum in Bydgoszcz, Nicolaus Copernicus University in Torun(妇产科与肿瘤学系,比多斯茨医学院,尼古拉斯·哥白尼大学图努恩分校)
Text2Earth: Unlocking Text-driven Remote Sensing Image Generation with a Global-Scale Dataset and a Foundation Model
Beihang University(北京航空航天大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shen Yuan Honors College of Beihang University(北京航空航天大学沈元荣誉学院);National University of Singapore(新加坡国立大学)
Rethinking Infrared Small Target Detection: A Foundation-Driven Efficient Paradigm
Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences(光电信息处理重点实验室,中国科学院);Shenyang Institute of Automation, Chinese Academy of Sciences(沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Sun Yat-sen University(中山大学);Tencent(腾讯);Nankai University(南开大学);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab)
Foundation Model-driven Key Anatomy Frame Selection for Blind-sweep Ultrasound Fetal Birth Weight Estimation
Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University(深圳大学医学超声图像计算实验室);School of Biomedical Engineering, Shenzhen University(深圳大学生物医学工程学院);School of Biomedical Engineering and Informatics, Nanjing Medical University(南京医科大学生物医学工程与信息学院);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心);Shandong Provincial Maternal and Child Health Care Hospital Affiliated to Qingdao University(青岛大学附属山东省妇幼保健院);Jiangxi Maternal and Child Health Hospital(江西省妇幼保健院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院)
FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentation
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Information Science and Engineering, Wuhan University of Science and Technology(武汉科技大学信息科学与工程学院);School of Computer Science, Wuhan University(武汉大学计算机学院)
Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models
National Institute of Health Data Science, Peking University(北京大学健康数据科学研究所);Institute of Medical Technology, Peking University(北京大学医学技术研究所);Deepwise Co., Ltd.(深度智医科技有限公司);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所)
Information-driven Fusion of Pathology Foundation Models for Enhanced Disease Characterization
Case Western Reserve University(凯斯西储大学);Department of Biomedical Engineering(生物医学工程系);Cleveland Clinic(克利夫兰诊所);Department of Pathology(病理学系);Emory University(埃默里大学);Department of Pediatrics(儿科学系);Louis Stokes VA Cleveland Medical Center(路易斯·斯托克斯退伍军人医疗中心)
SAM-Driven Weakly Supervised Nodule Segmentation with Uncertainty-Aware Cross Teaching
School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);University of Electronic Science and Technology of China(电子科技大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Xiangya Hospital, Central South University(中南大学湘雅医院);Hunan Frontline Medical Technology Co., Ltd(湖南前沿医疗科技有限公司)
Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation
University of Central Florida, Orlando, USA(佛罗里达大学);University College London, London, UK(伦敦大学学院);University of Illinois Urbana-Champaign, Champaign, USA(伊利诺伊大学厄巴纳-香槟分校);University of Trento, Trento, Italy(特伦托大学);The Hong Kong Polytechnic University, China(香港理工大学);University of Pennsylvania, Philadelphia, USA(宾夕法尼亚大学)
Medical SAM3通过微调SAM3模型,提升医学图像分割的通用提示驱动能力,解决领域偏移和复杂结构推理问题。
Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation
Peking University(北京大学);Hohai University(河海大学);Beijing Normal University-Hong Kong Baptist University United International College(北京师范大学-香港 Baptist大学联合国际学院);National Institute of Health Data Science, Peking University(健康数据科学国家研究院,北京大学);Institute of Medical Technology, Peking University(北京大学医学技术研究院);State Key Laboratory of General Artificial Intelligence, Peking University(通用人工智能国家重点实验室,北京大学)
Foundation Feature-Driven Online End-Effector Pose Estimation: A Marker-Free and Learning-Free Approach
Peking University(北京大学);Center on Frontiers of Computing Studies, School of Computer Science, Peking University(北京大学计算机科学学院计算前沿研究中心);PKU-Agibot Lab(北大-星猿哲实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机科学学院多媒体信息处理国家重点实验室)
Backdoor Attacks on Prompt-Driven Video Segmentation Foundation Models
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Fujian Normal University(福建师范大学);Jinan University(暨南大学)
Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration
Department of Artificial Intelligence, Chung-Ang University(Chung-Ang 大学人工智能系);Department of Statistics, Korea University(韩国大学统计系);School of Computer Science, University of Birmingham(伯明翰大学计算机科学学院);School of Computing, Gachon University(加成大学计算学院)
MedIQA: A Scalable Foundation Model for Prompt-Driven Medical Image Quality Assessment
Faculty of Applied Sciences, Macao Polytechnic University, Macao, China(澳门理工学院);Department of Engineer Science, University of Oxford, Oxford, UK(牛津大学工程科学系);Great Bay University, Dongguan, China(东莞大湾大学);College of Physics and Information Engineering, Fuzhou University, Fuzhou, China(福州大学物理与信息工程学院);Shanghai Jiao Tong University, Shanghai, China(上海交通大学);Department of Radiology, Shenzhen People’s Hospital, Shenzhen, China(深圳人民医院放射科)
GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Peking University(北京大学);The Hong Kong University of Science and Technology(香港科技大学)
Molecular-driven Foundation Model for Oncologic Pathology
Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);Cancer Program, Broad Institute of Harvard and MIT(哈佛与麻省理工学院博德研究所癌症项目);Health Sciences and Technology, Harvard-MIT(哈佛-麻省理工健康科学与技术部);Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院);Helmholtz Munich – German Research Center for Environment and Health(亥姆霍兹慕尼黑中心(德国环境与健康研究中心));School of Computation, Information and Technology, TUM(慕尼黑工业大学计算、信息与技术学院);Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT)(麻省理工学院电气工程与计算机科学系);Sydney Precision Data Science Center, The University of Sydney(悉尼大学悉尼精准数据科学中心);Department of Oncologic Pathology, Dana-Farber Cancer Institute(达纳-法伯癌症研究所肿瘤病理科);Department of Pathology, Boston Children’s Hospital(波士顿儿童医院病理科);Harvard Data Science Initiative, Harvard University(哈佛大学数据科学倡议)
A Synthetic Data-Driven Radiology Foundation Model for Pan-tumor Clinical Diagnosis
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Lab(上海人工智能实验室);The First Hospital of China Medical University(中国医科大学附属第一医院);Harvard Medical School(哈佛医学院)
Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains
School of Mechanical Engineering, Hubei University of Technology(湖北工业大学机械工程学院);Hubei Key Laboratory of Modern Manufacturing Quality Engineering, Hubei University of Technology(湖北工业大学现代制造质量工程重点实验室);School of Mechanical Science and Engineering, Huazhong University of Science and Technology(华中科技大学机械科学与工程学院)
Towards Spatial Transcriptomics-driven Pathology Foundation Models
Department of Pathology, Mass General Brigham, Harvard Medical School, Boston, MA, USA(病理学系,马萨诸塞州总医院与哈佛医学院,波士顿,马萨诸塞州,美国);Department of Computer Science & Technology, University of Cambridge, Cambridge, UK(计算机科学与技术系,剑桥大学,剑桥,英国);Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA(癌症计划,哈佛与麻省理工联合学院,剑桥,马萨诸塞州,美国);Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA(数据科学计划,达纳-法伯癌症研究所,波士顿,马萨诸塞州,美国);Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, MA, USA(哈佛-麻省理工健康科学与技术 division,麻省理工学院,剑桥,马萨诸塞州,美国);Télécom Paris, Institut Polytechnique de Paris, Paris, France(巴黎电信学院,巴黎理工学院,巴黎,法国);Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA(哈佛大学数据科学倡议,哈佛大学,剑桥,马萨诸塞州,美国)
Full end-to-end diagnostic workflow automation of 3D OCT via foundation model-driven AI for retinal diseases
Sun Yat-sen University(中山大学);Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);The Hong Kong Polytechnic University(香港理工大学);Zhongshan City People's Hospital(中山市人民医院);Guangdong Medical University(广东医科大学);The Third Affiliated Hospital of Sun Yat-sen University(中山大学附属第三医院);Zhaoqing Gaoyao People’s Hospital(肇庆高要人民医院);Tianjin Medical University Eye Hospital(天津医科大学眼科医院);Capital Medical University(首都医科大学);Beijing Tongren Hospital(北京同仁医院)
Towards Blind Bitstream-corrupted Video Recovery via a Visual Foundation Model-driven Framework
School of EEE, Nanyang Technological University(南洋理工大学电子工程系);School of EIC, Huazhong University of Science and Technology(华中科技大学电子信息学院);Dept. of EEE, The Hong Kong Polytechnic University(香港理工大学电子工程系)