Adapting Segment Anything Model 3 for Concept-Driven Lesion Segmentation in Medical Images: An Experimental Study
The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心放射肿瘤学系医学人工智能与自动化（MAIA）实验室）；Medical Image Processing Group, Department of Radiology, University of Pennsylvania（宾夕法尼亚大学放射学系医学图像处理组）；Division of Digestive and Liver Diseases, Department of Internal Medicine, University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心内科学系消化与肝脏疾病科）
Guoping Xu, Jayaram K. Udupa, Yubing Tong, Xin Long, Ying Zhang, Jie Deng, Weiguo Lu, You Zhang
准确的病变分割对医学图像分析至关重要，但现有方法多针对特定解剖部位或成像模态，限制了通用性。最近的视觉-语言基础模型使自然图像的概念驱动分割成为可能，为更灵活的医学图像分析提供了新方向。然而，基于概念提示的病变分割，尤其是使用最新版Segment Anything Model 3（SAM3）仍处于探索阶段。本文系统评估了SAM3在病变分割中的表现，通过几何边界框和基于概念的文本及图像提示，在多模态数据（包括多参数MRI、CT、超声、皮肤镜和内窥镜）中评估其性能。为提高鲁棒性，我们整合了额外的先验知识，如相邻切片预测、多参数信息和先验标注。进一步比较了不同的微调策略，包括部分模块微调、基于适配器的方法和全模型优化。在13个涵盖11种病变类型的数据集上，实验表明SAM3实现了强大的跨模态泛化能力、可靠的基于概念的分割以及精确的病变界定。这些结果突显了基于概念的基础模型在可扩展和实用医学图像分割中的潜力。代码和训练模型将在https://github.com/apple1986/lesion-sam3发布。
英文摘要：
Accurate lesion segmentation is essential in medical image analysis, yet most existing methods are designed for specific anatomical sites or imaging modalities, limiting their generalizability. Recent vision-language foundation models enable concept-driven segmentation in natural images, offering a promising direction for more flexible medical image analysis. However, concept-prompt-based lesion segmentation, particularly with the latest Segment Anything Model 3 (SAM3), remains underexplored.
In this work, we present a systematic evaluation of SAM3 for lesion segmentation. We assess its performance using geometric bounding boxes and concept-based text and image prompts across multiple modalities, including multiparametric MRI, CT, ultrasound, dermoscopy, and endoscopy. To improve robustness, we incorporate additional prior knowledge, such as adjacent-slice predictions, multiparametric information, and prior annotations. We further compare different fine-tuning strategies, including partial module tuning, adapter-based methods, and full-model optimization.
Experiments on 13 datasets covering 11 lesion types demonstrate that SAM3 achieves strong cross-modality generalization, reliable concept-driven segmentation, and accurate lesion delineation. These results highlight the potential of concept-based foundation models for scalable and practical medical image segmentation. Code and trained models will be released at: https://github.com/apple1986/lesion-sam3
ViPSAM: Visual Prompting Medical Image Segmentation Using Segment Anything Model
Department of Artificial Intelligence, Sungkyunkwan University, Republic of Korea（人工智能系，成均馆大学，大韩民国）；Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Republic of Korea（放射肿瘤科，三星医疗中心，成均馆大学医学院，大韩民国）；Department of MetaBioHealth, Sungkyunkwan University, Republic of Korea（元生物健康系，成均馆大学，大韩民国）
MedSAM2: Segment Anything in 3D Medical Images and Videos
University Health Network（多伦多大学健康网络）；Vector Institute（向量研究院）；Harvard Medical School（哈佛医学院）；Harvard University（哈佛大学）；University of Toronto（多伦多大学）；Peter Munk Cardiac Centre（彼得·芒克心脏中心）；AI Hub（人工智能中心）
提出MedSAM2，通过微调Segment Anything Model 2构建3D医学图像和视频分割基础模型，结合人在回路流程显著降低标注成本并提升分割性能。
Ryerson University（瑞尔森大学）；University of Toronto（多伦多大学）；Sunnybrook Health Sciences Centre（桑尼布鲁克健康科学中心）；St. Michael’s Hospital（圣迈克尔医院）；Keenan Research Centre for Biomedical Science（基南生物医学科学研究中心）；Institute for Biomedical Engineering, Science, and Technology (iBEST)（生物医学工程、科学与技术研究所（iBEST））
One of the key limitations in machine learning models is poor performance on data that is out of the domain of the training distribution. This is especially true for image analysis in magnetic resonance (MR) imaging, as variations in hardware and software create non-standard intensities, contrasts, and noise distributions across scanners. Recently, image translation models have been proposed to augment data across domains to create synthetic data points. In this paper, we investigate the application an unsupervised image translation model to augment MR images from a source dataset to a target …
Mask-Enhanced Segment Anything Model for Tumor Lesion Semantic Segmentation
Beihang University（北京航空航天大学）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；Peking University Cancer Hospital & Institute（北京大学肿瘤医院暨研究所）；Nanjing University（南京大学）
MOIS-SAM2: Exemplar-based Segment Anything Model 2 for multilesion interactive segmentation of neurofibromas in whole-body MRI
Institute for Applied Medical Informatics, Institute of Computational Neuroscience, and Center for Biomedical Artificial Intelligence (bAIome), University Medical Center Hamburg-Eppendorf（应用医学信息学研究所、计算神经科学研究所和生物医学人工智能中心（bAIome）、汉堡医学院中心）；Department of Diagnostic and Interventional Radiology and Nuclear Medicine, University Medical Center Hamburg-Eppendorf（诊断与介入放射学及核医学系、汉堡医学院中心）；Department of Neurology, University Medical Center Hamburg-Eppendorf（神经病学系、汉堡医学院中心）
MOIS-SAM2是一种基于示例的Segment Anything Model 2，用于全身体部MRI中神经纤维瘤多病灶交互分割，实现了高精度和可扩展性，支持临床应用。
Part-aware Prompted Segment Anything Model for Adaptive Segmentation
Department of Computer Science and Engineering, University of Michigan（计算机科学与工程系，密歇根大学）；Department of Electrical and Computer Engineering, University of Michigan（电气与计算机工程系，密歇根大学）
In this work, we address the TrackRAD2025 challenge of real-time tumor tracking in cine-MRI sequences of the thoracic and abdominal regions under strong data scarcity constraints. Two complementary strategies were explored: (i) unsupervised registration with the IMPACT similarity metric and (ii) foundation model-based segmentation leveraging SAM 2.1 and its recent variants through prompt-based interaction. Due to the one-second runtime constraint, the SAM-based method was ultimately selected. The final configuration used SAM2.1 b+ with mask-based prompts from the first annotated slice, fine-tu…
Adapting a Segmentation Foundation Model for Medical Image Classification
Department of Computer Science, University of Texas Rio Grande Valley（德克萨斯理工大学里奥格兰德分校计算机科学系）；Department of Computer Science, Texas Woman’s University（德克萨斯女性大学计算机科学系）；Sewickley Academy（塞维克利学院）；Department of Mathematical Sciences, The University of Texas at Dallas（德克萨斯大学达拉斯分校数学科学系）；Department of Computer Science and Engineering, University of Notre Dame（诺特丹大学计算机科学与工程系）
本文提出将 SAM 图像编码器作为冻结特征提取器，并设计空间局部化通道注意力机制 SLCA，将其集成到深度学习分类模型中，以增强对关键区域的关注，在三个公开医学图像分类数据集上验证了方法的有效性和数据高效性。
Information and Computer Science, King Fahd University of Petroleum and Minerals（信息与计算机科学系，国王法赫德石油与矿物大学）；Faculty of Science and Information Technology, Daffodil International University（科学与信息技术学院，达福尔国际大学）；Department of Human Anatomy and Physiology, The Faculty of Health Sciences, University of Johannesburg（人类解剖学与生理学系，健康科学学院，约翰内斯堡大学）；Centre for Data Analytics（数据分析中心）；School of Psychology, Bond University（心理学学院，邦德大学）
Context: Mpox is a zoonotic disease caused by the Mpox virus, which shares similarities with other skin conditions, making accurate early diagnosis challenging. Artificial intelligence (AI), especially Deep Learning (DL), has a strong tool for medical image analysis; however, pre-trained models like CNNs and XAI techniques for mpox detection is underexplored. Objective: This study aims to evaluate the effectiveness of pre-trained CNN models (VGG16, VGG19, InceptionV3, MobileNetV2) for the early detection of monkeypox using binary and multi-class datasets. It also seeks to enhance model interpr…
SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens
School of Electrical Engineering and Computer Science, The University of Queensland（电气工程与计算机科学学院，昆士兰大学）；School of Engineering, College of Engineering, Science and Environment, University of Newcastle（工程学院，工程、科学与环境学院，纽卡斯尔大学）；School of Human Movement and Nutrition Sciences, The University of Queensland（人类运动与营养科学学院，昆士兰大学）
Towards Label-Free Brain Tumor Segmentation: Unsupervised Learning with Multimodal MRI
Universitat Politècnica de Catalunya (UPC)（西班牙巴塞罗那理工大学）；University College Dublin (UCD)（都柏林大学）；Dublin City University (DCU)（都柏林城市大学）；Insight Research Ireland Center For Data Analytics（爱尔兰数据分析研究中心）
GlanceSeg: Real-time microaneurysm lesion segmentation with gaze-map-guided foundation model for early detection of diabetic retinopathy
Southern University of Science and Technology（南方科技大学）；Wenzhou Medical University（温州医科大学）；Intelligent Vision Plus Technology Co., Ltd.（深圳市智能视觉加科技有限公司）；The Chinese University of Hong Kong（香港中文大学）
McGill University（麦吉尔大学）；MILA Quebec AI Institute（魁北克人工智能研究所MILA）
Large, annotated datasets are not widely available in medical image analysis due to the prohibitive time, costs, and challenges associated with labelling large datasets. Unlabelled datasets are easier to obtain, and in many contexts, it would be feasible for an expert to provide labels for a small subset of images. This work presents an information-theoretic active learning framework that guides the optimal selection of images from the unlabelled pool to be labeled based on maximizing the expected information gain (EIG) on an evaluation dataset. Experiments are performed on two different medic…
University of Isfahan（伊斯法罕大学）；Duke University（杜克大学）；Duke University Medical Center（杜克大学医学中心）
Optical coherence tomography (OCT) helps ophthalmologists assess macular edema, accumulation of fluids, and lesions at microscopic resolution. Quantification of retinal fluids is necessary for OCT-guided treatment management, which relies on a precise image segmentation step. As manual analysis of retinal fluids is a time-consuming, subjective, and error-prone task, there is increasing demand for fast and robust automatic solutions. In this study, a new convolutional neural architecture named RetiFluidNet is proposed for multi-class retinal fluid segmentation. The model benefits from hierarchi…
LCE: A Framework for Explainability of DNNs for Ultrasound Image Based on Concept Discovery
School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；The Third People’s Hospital of Chengdu, Affiliated Hospital of Southwest Jiaotong University（西南交通大学附属医院成都市第三人民医院）
Towards SAMBA: Segment Anything Model for Brain Tumor Segmentation in Sub-Sharan African Populations
Nile University（尼罗大学）；Friedrich-Alexander-Universität（埃尔朗根-纽伦堡大学）；Makerere University（马凯雷雷大学）；Malawi University of Science and Technology（马拉维科技大学）；Medical Artificial Intelligence Laboratory (MAI Lab)（医学人工智能实验室（MAI Lab））；Lawson Health Research Institute（劳森健康研究院）；University of British Columbia（不列颠哥伦比亚大学）；Montreal Neurological Institute, McGill University（麦吉尔大学蒙特利尔神经学研究所）；University of Cape Town（开普敦大学）
Federated Learning with Partially Labeled Data: A Conditional Distillation Approach
Department of Computer Science and Information Engineering, National Taiwan University（计算机科学与信息工程系，国立台湾大学）；Graduate School of Informatics, Nagoya University（资讯研究生院，名古屋大学）；Institute of Applied Mathematical Sciences, National Taiwan University（应用数学科学研究所，国立台湾大学）；NVIDIA Corporation（NVIDIA公司）
City University of Hong Kong（香港城市大学）；University College London（伦敦大学学院）
Dice similarity coefficient (DSC) and Hausdorff distance (HD) are widely used for evaluating medical image segmentation. They have also been criticised, when reported alone, for their unclear or even misleading clinical interpretation. DSCs may also differ substantially from HDs, due to boundary smoothness or multiple regions of interest (ROIs) within a subject. More importantly, either metric can also have a nonlinear, non-monotonic relationship with outcomes based on Type 1 and 2 errors, designed for specific clinical decisions that use the resulting segmentation. Whilst cases causing disagr…
From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images
Carnegie Mellon University（卡内基梅隆大学）；Industrial University of Ho Chi Minh City（胡志明市工业大学）；University of North Carolina at Charlotte（北卡罗来纳州夏洛特大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
UCL（伦敦大学学院）；UCLH NHS Foundation Trust（伦敦大学学院医院NHS基金会信托）
The need for training data can impede the adoption of novel imaging modalities for learning-based medical image analysis. Domain adaptation methods partially mitigate this problem by translating training data from a related source domain to a novel target domain, but typically assume that a one-to-one translation is possible. Our work addresses the challenge of adapting to a more informative target domain where multiple target samples can emerge from a single source sample. In particular we consider translating from mp-MRI to VERDICT, a richer MRI modality involving an optimized acquisition pr…
Performance of convolutional neural networks (CNNs) in image analysis tasks is often marred in the presence of acquisition-related distribution shifts between training and test images. Recently, it has been proposed to tackle this problem by fine-tuning trained CNNs for each test image. Such test-time-adaptation (TTA) is a promising and practical strategy for improving robustness to distribution shifts as it requires neither data sharing between institutions nor annotating additional data. Previous TTA methods use a helper model to increase similarity between outputs and/or features extracted …
SAM2-SGP: Enhancing SAM2 for Medical Image Segmentation via Support-Set Guided Prompting
J. Crayton Pruitt Family Department of Biomedical Engineering University of Florida；Department of Electrical \& Computer Engineering University of Florida
