Vision foundation models are renowned for the generalization ability due to massive training data. Nevertheless, they demand tremendous training resources, and the training data is often inaccessible, e.g., CLIP, DINOv2, posing great challenges to developing derivatives that could facilitate the research. In this work, we offer a very simple and general solution, named \textit{Proteus}, to distill foundation models into smaller equivalents on ImageNet-1K without access to the original training data. Specifically, we remove the designs from conventional knowledge distillation settings that result in dataset bias and present three levels of training objectives, i.e., token, patch, and feature, to maximize the efficacy of knowledge transfer. In this manner, Proteus is trained at ImageNet-level costs with surprising ability, facilitating the accessibility of training foundation models for the broader research community. When leveraging DINOv2-g/14 as the teacher, Proteus-L/14 matches the performance of the Oracle method DINOv2-L/14 (142M training data) across 19 benchmarks and outperforms other vision foundation models including CLIP-L/14 (400M), OpenCLIP-L/14 (400M/2B) and SynCLR-L/14 (600M) with a significantly smaller training set of 1.2M images.
Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing
IBM Research（IBM研究院）；The University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；University Space Research Association (USRA)（大学空间研究协会）；NASA Goddard Space Flight Center（美国国家航空航天局戈达德太空飞行中心）；SETI Institute（搜寻地外文明研究所）；NASA Ames Research Center（美国国家航空航天局艾姆斯研究中心）；University of Maryland, Baltimore County (UMBC)（马里兰大学巴尔的摩县分校）；Howard University（霍华德大学）
Exemplar-free incremental learning is extremely challenging due to inaccessibility of data from old tasks. In this paper, we attempt to exploit the knowledge encoded in a previously trained classification model to handle the catastrophic forgetting problem in continual learning. Specifically, we introduce a so-called knowledge delegator, which is capable of transferring knowledge from the trained model to a randomly re-initialized new model by generating informative samples. Given the previous model only, the delegator is effectively learned using a self-distillation mechanism in a data-free m…
Is Self-Supervision Enough? Benchmarking Foundation Models Against End-to-End Training for Mitotic Figure Classification
Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）；University of Veterinary Medicine, Vienna（维也纳兽医大学）；Center for AI and Data Science, Julius-Maximilians-Universität Würzburg（维尔茨堡大学朱利叶斯-马克西米利安人工智能与数据科学中心）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Flensburg University of Applied Sciences（弗伦斯堡应用技术大学）
Baseline Method of the Foundation Model Challenge for Ultrasound Image Analysis
The First Affiliated Hospital of Jinan University（暨南大学第一附属医院）；Southern Medical University（南方医科大学）；Guangzhou Women and Children's Medical Center（广州市妇女儿童医疗中心）
Foundation Model Guided Dual-Branch Co-Adaptation for Source-Free EEG Decoding
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；College of Artifical Intelligence and Automation, Hohai University（河海大学人工智能与自动化学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；Brainnetcome Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所脑网络中心）；Information Systems Technology and Design, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计）
Comparing ImageNet Pre-training with Digital Pathology Foundation Models for Whole Slide Image-Based Survival Analysis
Imperial College London, Department of Electrical and Electronic Engineering, London, UK（帝国理工学院伦敦分校，电子与电气工程系）；Department of Pathology, Institut de Pathologie Multisite, Groupement Hospitalier Sud, Lyon University Hospital, Pierre-Bénite, France（病理学系，多站点病理研究所，南部医院集团，里昂大学医院，皮埃尔-贝内蒂，法国）；University of Lyon, Université Claude Bernard Lyon 1, Lyon, France（里昂大学， Claude Bernard 里昂第一大学，里昂，法国）
SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation
Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Spain（智能技术研究中心（CiTIUS），圣地亚哥-德孔波斯特拉大学，西班牙）；Departmento de Electrónica e Computación, Universidade de Santiago de Compostela, Spain（电子与计算系，圣地亚哥-德孔波斯特拉大学，西班牙）；IBM Research Europe, Zurich, Switzerland（IBM欧洲研究院，瑞士苏黎世）
University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；Max-Planck-Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）
Machine learning models have been found to learn shortcuts -- unintended decision rules that are unable to generalize -- undermining models' reliability. Previous works address this problem under the tenuous assumption that only a single shortcut exists in the training data. Real-world images are rife with multiple visual cues from background to texture. Key to advancing the reliability of vision systems is understanding whether existing methods can overcome multiple shortcuts or struggle in a Whac-A-Mole game, i.e., where mitigating one shortcut amplifies reliance on others. To address this s…
Rutgers University（罗格斯大学）；Amazon Web Services（亚马逊云科技）
Models should be able to adapt to unseen data during test-time to avoid performance drops caused by inevitable distribution shifts in real-world deployment scenarios. In this work, we tackle the practical yet challenging test-time adaptation (TTA) problem, where a model adapts to the target domain without accessing the source data. We propose a simple recipe called \textit{Data-efficient Prompt Tuning} (DePT) with two key ingredients. First, DePT plugs visual prompts into the vision Transformer and only tunes these source-initialized prompts during adaptation. We find such parameter-efficient …
UltraSam: A Foundation Model for Ultrasound using Large Open-Access Segmentation Datasets
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，斯特拉斯堡，法国）；IHU-Strasbourg, Institute of Image-Guided Surgery, Strasbourg, France（斯特拉斯堡IHU，影像引导手术研究所，斯特拉斯堡，法国）
Learning to synthesize data has emerged as a promising direction in zero-shot quantization (ZSQ), which represents neural networks by low-bit integer without accessing any of the real data. In this paper, we observe an interesting phenomenon of intra-class heterogeneity in real data and show that existing methods fail to retain this property in their synthetic images, which causes a limited performance increase. To address this issue, we propose a novel zero-shot quantization method referred to as IntraQ. First, we propose a local object reinforcement that locates the target objects at differe…
While post-training quantization receives popularity mostly due to its evasion in accessing the original complete training dataset, its poor performance also stems from scarce images. To alleviate this limitation, in this paper, we leverage the synthetic data introduced by zero-shot quantization with calibration dataset and propose a fine-grained data distribution alignment (FDDA) method to boost the performance of post-training quantization. The method is based on two important properties of batch normalization statistics (BNS) we observed in deep layers of the trained network, (i.e.), inter-…
Foundation Models for Slide-level Cancer Subtyping in Digital Pathology
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano (HUMAN-Tech)（以人为本技术大学研究院（HUMAN-Tech））；Universitat Politécnica de Valencia（瓦伦西亚理工大学）；Valencian Graduate School and Research Network for Artificial Intelligence (valgraAI)（瓦伦西亚人工智能研究生院与研究网络（valgraAI））
Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts
Technical University of Munich (TUM)（慕尼黑工业大学）；German Aerospace Center (DLR)（德国航空航天中心）；Massachusetts Institute of Technology (MIT)（麻省理工学院）；Taylor Geospatial（泰勒地理空间公司）
University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）
The goal of this work is to segment and name regions of images without access to pixel-level labels during training. To tackle this task, we construct segmenters by distilling the complementary strengths of two foundation models. The first, CLIP (Radford et al. 2021), exhibits the ability to assign names to image content but lacks an accessible representation of object structure. The second, DINO (Caron et al. 2021), captures the spatial extent of objects but has no knowledge of object names. Our method, termed NamedMask, begins by using CLIP to construct category-specific archives of images. …
Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging
Cluster of Excellence “Machine Learning”（卓越集群“机器学习”）；University of Tübingen（蒂宾根大学）；Faculty of Health Sciences and Medicine, University of Lucerne（卢塞恩大学健康科学与医学院）
Microsoft Cloud and AI（微软云与人工智能部门）；Microsoft Research Redmond（微软雷德蒙德研究院）
Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP, ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new co…
Overcoming Data Scarcity in Biomedical Imaging with a Foundational Multi-Task Model
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所）；Institute of Image Analysis and Computer Vision, Faculty of Informatics and Data Science, University of Regensburg（雷根斯堡大学信息学与数据科学学院图像分析与计算机视觉研究所）；Institute for Pathology, Hannover Medical School（汉诺威医学院病理学研究所）；Institute for Neuropathology, University Clinic Freiburg（弗莱堡大学医院神经病理学研究所）；Institute for Experimental Molecular Imaging, RWTH Aachen University（亚琛工业大学实验分子影像学研究所）
A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction
Tandon School of Engineering, New York University（纽约大学坦登工程学院）；NYU Langone Health, New York University（纽约大学朗格尼健康中心）；New York University Abu Dhabi（纽约大学阿布扎比分校）
