Vision-Language Models (VLMs) have shown impressive performance in vision tasks, but adapting them to new domains often requires expensive fine-tuning. Prompt tuning techniques, including textual, visual, and multimodal prompting, offer efficient alternatives by leveraging learnable prompts. However, their application to Vision-Language Segmentation Models (VLSMs) and evaluation under significant domain shifts remain unexplored. This work presents an open-source benchmarking framework, TuneVLSeg, to integrate various unimodal and multimodal prompt tuning techniques into VLSMs, making prompt tuning usable for downstream segmentation datasets with any number of classes. TuneVLSeg includes $6$ prompt tuning strategies on various prompt depths used in $2$ VLSMs totaling of $8$ different combinations. We test various prompt tuning on $8$ diverse medical datasets, including $3$ radiology datasets (breast tumor, echocardiograph, chest X-ray pathologies) and $5$ non-radiology datasets (polyp, ulcer, skin cancer), and two natural domain segmentation datasets. Our study found that textual prompt tuning struggles under significant domain shifts, from natural-domain images to medical data. Furthermore, visual prompt tuning, with fewer hyperparameters than multimodal prompt tuning, often achieves performance competitive to multimodal approaches, making it a valuable first attempt. Our work advances the understanding and applicability of different prompt-tuning techniques for robust domain-specific segmentation. The source code is available at https://github.com/naamiinepal/tunevlseg.
DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation
School of Electrical Engineering, Tel Aviv University, Israel(特拉维夫大学电气工程学院);School of Biomedical Engineering, Tel Aviv University, Israel(特拉维夫大学生物医学工程学院)
Disentangling Prompt Dependence to Evaluate Segmentation Reliability in Gynecological MRI
Laboratoire Traitement du Signal et de l’Image (LTSI, INSERM UMR 1099), Université de Rennes, Rennes, France(信号与图像处理实验室(LTSI,INSERM UMR 1099),雷恩大学,法国雷恩);Centre Hospitalier Universitaire de Rennes (CHU Rennes), Rennes, France(雷恩大学医院(CHU Rennes),法国雷恩)
Taiyuan University of Technology(太原理工大学);University of Science and Technology of China(中国科学技术大学);Second Hospital of Shanxi Medical University(山西医科大学第二医院)
PromptForSegCXR: Prompt-Driven Multi-Organ and Multi-Disease Segmentation in Chest X-rays using a Multi-stage Fusion Mechanism
1 ,5Department of Computer Science \& Engineering,\ University of Engineering \& Technology, Rajshahi-6204, Bangladesh;2 Department of Electrical;Electronic Engineering,\ University of Engineering;Bachelor of Surgery,\ Suhrawardy Medical College , Sher-E-Bangla Nagar, Dhaka-1207;6 Department of Biomedical Engineering,\ University of Engineering
Part-aware Prompted Segment Anything Model for Adaptive Segmentation
Department of Computer Science and Engineering, University of Michigan(计算机科学与工程系,密歇根大学);Department of Electrical and Computer Engineering, University of Michigan(电气与计算机工程系,密歇根大学)
SLIP: Segmentation with Low-latency Interactive Prompting for 3D Medical Images
IRCAD France(法国国际消化道疾病研究中心);Université de Strasbourg, CNRS, ICube(斯特拉斯堡大学、法国国家科学研究中心、ICube实验室);IRCCS Humanitas Research Hospital(圣心研究医院);Humanitas University(圣心大学);Fondazione Policlinico Universitario A. Gemelli, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS)(圣心综合大学医院)
Promptable cancer segmentation using minimal expert-curated data
Department of Medical Physics and Biomedical Engineering, University College London(医学物理与生物医学工程系,伦敦大学学院);UCL Hawkes Institute, University College London(UCL Hawkes研究所,伦敦大学学院);Division of Surgery and Interventional Science, University College London(外科与介入科学系,伦敦大学学院);Department of Radiology(放射科;泌尿科,斯坦福大学);and Department of Urology, Stanford University
Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization
School of Information and Communication Technology, Griffith University, QLD, Australia(信息与通信技术学院,格里菲斯大学,昆士兰,澳大利亚);School of MDP -- Clinical Medicine, Griffith University, QLD, Australia(MDP临床医学学院,格里菲斯大学,昆士兰,澳大利亚)
UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images
Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心);Chinese University of Hong Kong(香港中文大学)
ViPSAM: Visual Prompting Medical Image Segmentation Using Segment Anything Model
Department of Artificial Intelligence, Sungkyunkwan University, Republic of Korea(人工智能系,成均馆大学,大韩民国);Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Republic of Korea(放射肿瘤科,三星医疗中心,成均馆大学医学院,大韩民国);Department of MetaBioHealth, Sungkyunkwan University, Republic of Korea(元生物健康系,成均馆大学,大韩民国)
OBSeg: Accurate and Fast Instance Segmentation Framework Using Segmentation Foundation Models with Oriented Bounding Box Prompts
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Science, China(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院,中国);School of Artificial Intelligence, University of Chinese Academy of Science, China(人工智能学院,中国科学院大学,中国);Department of Radiology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing(放射科,北医三院,北京医科大学,中国医学科学院,北京)
Insight Any Instance: Promptable Instance Segmentation for Remote Sensing Images
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);University of Chinese Academy of Sciences(中国科学院大学);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院)
IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment
University of Chinese Academy of Sciences(中国科学院大学);Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences(中国科学院空间应用工程与技术中心)
Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectification
Artificial Intelligence Innovation and Incubation Institute, Fudan University(复旦大学人工智能创新与孵化研究院);Shanghai Academy of Artificial Intelligence for Science(上海科学人工智能研究院)
SRPR-Net: Semantic and Relational Prompt Refinement for Automated SAM-based Instance Segmentation
Ruijin-XJTLU Intelligent Medicine Institute, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine(瑞金-西交利物浦智能医学研究所,瑞金医院,上海交通大学医学院);School of Biomedical Engineering, Shanghai Jiao Tong University(上海交通大学生物医学工程学院)
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院);University of Electronic Science and Technology of China(电子科技大学);Wuhan Vocational College of Software and Engineering(武汉软件工程职业学院)
Unleashing the Power of Prompt-driven Nucleus Instance Segmentation
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Engineering, Westlake University(西湖大学工学院);University of Liverpool(利物浦大学)
CPS4: Class Prompt driven Semi-Supervised Spine Segmentation with Class-specific Consistency Constraint
School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);Department of Nuclear Medicine, Shengjing Hospital of China Medical University(中国医科大学附属盛京医院核医学科);Department of Computer and Data Science, Case Western Reserve University(凯斯西储大学计算机与数据科学系);Department of Biomedical Engineering, Case Western Reserve University(凯斯西储大学生物医学工程系)