We propose a straightforward yet highly effective few-shot fine-tuning strategy for adapting the Segment Anything (SAM) to anatomical segmentation tasks in medical images. Our novel approach revolves around reformulating the mask decoder within SAM, leveraging few-shot embeddings derived from a limited set of labeled images (few-shot collection) as prompts for querying anatomical objects captured in image embeddings. This innovative reformulation greatly reduces the need for time-consuming online user interactions for labeling volumetric images, such as exhaustively marking points and bounding boxes to provide prompts slice by slice. With our method, users can manually segment a few 2D slices offline, and the embeddings of these annotated image regions serve as effective prompts for online segmentation tasks. Our method prioritizes the efficiency of the fine-tuning process by exclusively training the mask decoder through caching mechanisms while keeping the image encoder frozen. Importantly, this approach is not limited to volumetric medical images, but can generically be applied to any 2D/3D segmentation task. To thoroughly evaluate our method, we conducted extensive validation on four datasets, covering six anatomical segmentation tasks across two modalities. Furthermore, we conducted a comparative analysis of different prompting options within SAM and the fully-supervised nnU-Net. The results demonstrate the superior performance of our method compared to SAM employing only point prompts (approximately 50% improvement in IoU) and performs on-par with fully supervised methods whilst reducing the requirement of labeled data by at least an order of magnitude.
Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain. Contrastive learning (CL) in the context of UDA can help to better separate classes in feature space. However, in image segmentation, the large memory footprint due to the computation of the pixel-wise contrastive loss makes it prohibitive to use. Furthermore, labeled target data is not easily available in medical imaging, and obtaining new samples is not economical. As a result, in this work, we tackle a more challenging UDA task when there are only a few (fewsho…
In recent works, utilizing a deep network trained on meta-training set serves as a strong baseline in few-shot learning. In this paper, we move forward to refine novel-class features by finetuning a trained deep network. Finetuning is designed to focus on reducing biases in novel-class feature distributions, which we define as two aspects: class-agnostic and class-specific biases. Class-agnostic bias is defined as the distribution shifting introduced by domain difference, which we propose Distribution Calibration Module(DCM) to reduce. DCM owes good property of eliminating domain difference an…
VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness
Nanjing University（南京大学）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；Tsinghua University（清华大学）；Panasonic（松下电器（Panasonic））；University of California Berkeley（加州大学伯克利分校）
Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging
Georg-August-University Göttingen, Institute of Physics（哥廷根大学物理学院）；Georg-August-University Göttingen, Institute of Computer Science（哥廷根大学计算机科学学院）；CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen（下萨克森人工智能与医学因果方法中心）；Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen（多尺度生物成像集群（MBExC），哥廷根大学）
本文首次系统研究了面向生物医学图像分割的 SAM 参数高效微调方法，发现 PEFT 层的放置位置比层类型对效率影响更大，并提出了资源高效的微调方案。
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
Few-shot learning is challenging due to its very limited data and labels. Recent studies in big transfer (BiT) show that few-shot learning can greatly benefit from pretraining on large scale labeled dataset in a different domain. This paper asks a more challenging question: "can we use as few as possible labels for few-shot learning in both pretraining (with no labels) and fine-tuning (with fewer labels)?". Our key insight is that the clustering of target samples in the feature space is all we need for few-shot finetuning. It explains why the vanilla unsupervised pretraining (poor clustering) …
Joint-Optimized Unsupervised Adversarial Domain Adaptation in Remote Sensing Segmentation with Prompted Foundation Model
School of Electronics and Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）；University of Liverpool（利物浦大学）；Image Processing Center, School of Astronautics, Beihang University（北京航空航天大学宇航学院图像处理中心）
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Few-shot classification aims at classifying categories of a novel task by learning from just a few (typically, 1 to 5) labelled examples. An effective approach to few-shot classification involves a prior model trained on a large-sample base domain, which is then finetuned over the novel few-shot task to yield generalizable representations. However, task-specific finetuning is prone to overfitting due to the lack of enough training examples. To alleviate this issue, we propose a new finetuning approach based on contrastive learning that reuses unlabelled examples from the base domain in the for…
National University of Singapore（新加坡国立大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Tianjin University（天津大学）
Meta-learning has been proposed as a framework to address the challenging few-shot learning setting. The key idea is to leverage a large number of similar few-shot tasks in order to learn how to adapt a base-learner to a new task for which only a few labeled samples are available. As deep neural networks (DNNs) tend to overfit using a few samples only, meta-learning typically uses shallow neural networks (SNNs), thus limiting its effectiveness. In this paper we propose a novel few-shot learning method called meta-transfer learning (MTL) which learns to adapt a deep NN for few shot learning tas…
Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）；University of California, Berkeley（加州大学伯克利分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
