Segment Any Point Cloud Sequences by Distilling Vision Foundation Models
Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）
Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu
Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable 45.0% mIoU on nuScenes after linear probing, surpassing random initialization by 36.9% mIoU and outperforming prior arts by 6.1% mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets.
DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features
NVIDIA Research（英伟达研究院）；University of Toronto（多伦多大学）；University of Southern California（南加州大学）；National University of Singapore（新加坡国立大学）；Stanford University（斯坦福大学）
Deep audio representation learning using multi-modal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information…
University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）
The goal of this work is to segment and name regions of images without access to pixel-level labels during training. To tackle this task, we construct segmenters by distilling the complementary strengths of two foundation models. The first, CLIP (Radford et al. 2021), exhibits the ability to assign names to image content but lacks an accessible representation of object structure. The second, DINO (Caron et al. 2021), captures the spatial extent of objects but has no knowledge of object names. Our method, termed NamedMask, begins by using CLIP to construct category-specific archives of images. …
While fine-tuning pre-trained networks has become a popular way to train image segmentation models, such backbone networks for image segmentation are frequently pre-trained using image classification source datasets, e.g., ImageNet. Though image classification datasets could provide the backbone networks with rich visual features and discriminative ability, they are incapable of fully pre-training the target model (i.e., backbone+segmentation modules) in an end-to-end manner. The segmentation modules are left to random initialization in the fine-tuning process due to the lack of segmentation l…
From SAM to DINOv2: Towards Distilling Foundation Models to Lightweight Baselines for Generalized Polyp Segmentation
Malaviya National Institute of Technology, Jaipur, India（马利瓦国家理工学院，印度斋普尔）；Côte d’Azur University, France（蔚蓝海岸大学，法国）；University of South Dakota, USA（南达科他大学，美国）
MorphDistill: Distilling Unified Morphological Knowledge from Pathology Foundation Models for Colorectal Cancer Survival Prediction
Department of Pathology, College of Medicine, The Ohio State University Wexner Medical Center（俄亥俄州立大学医学学院病理学系，韦克斯纳医学中心）；Center for Artificial Intelligence Research, Wake Forest University School of Medicine（威克森林大学医学院人工智能研究中心）
From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than…
CellVTA: Enhancing Vision Foundation Models for Accurate Cell Segmentation and Classification
ShanghaiTech University（上海科技大学）；Shanghai Ocean University（上海海洋大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects
Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Computer Science（计算机科学系）；Department of Radiology（放射学系）；Department of Biostatistics & Bioinformatics（生物统计学与生物信息学系）；Minnesota Health Solutions（明尼苏达健康解决方案）
Distilling Expert Surgical Knowledge: How to train local surgical VLMs for anatomy explanation in Complete Mesocolic Excision
1 Institute of Medical Technology；Intelligent Systems, Hamburg University of Technology, Hamburg, Germany 2 Department of General, Visceral；Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
MSEG-VCUQ: Multimodal SEGmentation with Enhanced Vision Foundation Models, Convolutional Neural Networks, and Uncertainty Quantification for High-Speed Video Phase Detection Data
Can Foundation Models Really Segment Tumors? A Benchmarking Odyssey in Lung CT Imaging
College of Computer Science and Software Engineering, Shenzhen University（计算机科学与软件工程学院，深圳大学）；Department of Naval, Electrical, Electronics and Telecommunications Engineering, University of Genoa（海军、电气、电子与电信工程部门，热那亚大学）
AutoMiSeg: Automatic Medical Image Segmentation via Test-Time Adaptation of Foundation Models
Carnegie Mellon University（卡内基梅隆大学）；Brown University（布朗大学）；National Institute of Technology Karnataka（印度卡纳塔克国家理工学院）；University of Pittsburgh（匹兹堡大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping
Arizona State University（亚利桑那州立大学）；Woodwell Climate Research Center（伍德韦尔气候研究中心）；University of Connecticut（康涅狄格大学）；U.S. Geological Survey（美国地质调查局）；University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；University of Illinois at Urbana-Champaign（伊利诺伊大学香槟分校）
本文评估了以 Meta SAM 为代表的视觉基础模型在永久冻土地貌分割中的表现，通过多种提示策略和微调测试其零样本能力与领域适应性，并指出其在挑战性地理空间任务中仍需改进。
Fine-tuning vision foundation model for crack segmentation in civil infrastructures
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Civil Engineering, Tsinghua University（清华大学土木工程系）；School of Civil and Resource Engineering, University of Science and Technology Beijing（北京科技大学土木与资源工程学院）
SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
The foundation model is not the last chapter of the model production pipeline. Transferring with few data in a general way to thousands of downstream tasks is becoming a trend of the foundation model's application. In this paper, we proposed a universal transfer framework: One to Transfer All (OTA) to transfer any Vision Foundation Model (VFM) to any downstream tasks with few downstream data. We first transfer a VFM to a task-specific model by Image Re-representation Fine-tuning (IRF) then distilling knowledge from a task-specific model to a deployed model with data produced by Downstream Imag…
