The Segment Anything Model (SAM) has exhibited outstanding performance in various image segmentation tasks. Despite being trained with over a billion masks, SAM faces challenges in mask prediction quality in numerous scenarios, especially in real-world contexts. In this paper, we introduce a novel prompt-driven adapter into SAM, namely Prompt Adapter Segment Anything Model (PA-SAM), aiming to enhance the segmentation mask quality of the original SAM. By exclusively training the prompt adapter, PA-SAM extracts detailed information from images and optimizes the mask decoder feature at both sparse and dense prompt levels, improving the segmentation performance of SAM to produce high-quality masks. Experimental results demonstrate that our PA-SAM outperforms other SAM-based methods in high-quality, zero-shot, and open-set segmentation. We're making the source code and models available at https://github.com/xzz2/pa-sam.
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
University of Oxford（牛津大学）；National University of Singapore（新加坡国立大学）；University of Alberta（阿尔伯塔大学）；A*STAR（新加坡科技研究局）；Carnegie Mellon University（卡内基梅隆大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Hongkong（香港大学）
本文提出Medical SAM Adapter（Med-SA），通过轻量适配技术将医学知识融入SAM，利用SD-Trans和HyP-Adpt适配2D模型至3D医学图像，在17个任务上以仅2%参数更新超越SOTA方法。
SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentation
KOKONI, Moxin (Huzhou) Tech. Co., LTD（摩西（湖州）科技有限公司）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（新加坡南洋理工大学电子与电气工程学院）；College of Computing and Data Science, Nanyang Technological University（新加坡南洋理工大学计算与数据科学学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）
SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More
KOKONI, Moxin (Huzhou) Tech. Co., LTD（科科尼（湖州）魔芯科技有限公司）；School of Information Engineering, Huzhou University（湖州大学信息工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Information Systems Technology and Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计支柱学院）
Prompting Segment Anything Model with Domain-Adaptive Prototype for Generalizable Medical Image Segmentation
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；Medical Artificial Intelligence Research Institute of Renmin Hospital, Wuhan University（武汉大学人民医院医学人工智能研究院）；School of Computer Science, Central China Normal University（华中师范大学计算机学院）
SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More
KOKONI, Moxin (Huzhou) Tech. Co., LTD（莫西（湖州）科技有限公司KOKONI）；Beihang University（北京航空航天大学）；Zhejiang University（浙江大学）；Huzhou University（湖州师范学院）；Singapore University of Technology and Design（新加坡科技设计大学）
University at Buffalo（布法罗大学）；Duke University（杜克大学）
Large pre-trained vision-language (VL) models have shown significant promise in adapting to various downstream tasks. However, fine-tuning the entire network is challenging due to the massive number of model parameters. To address this issue, efficient adaptation methods such as prompt tuning have been proposed. We explore the idea of prompt tuning with multi-task pre-trained initialization and find it can significantly improve model performance. Based on our findings, we introduce a new model, termed Prompt-Adapter, that combines pre-trained prompt tunning with an efficient adaptation network…
Adapter-Enhanced Semantic Prompting for Continual Learning
Beijing Key Laboratory of Multimedia and Intelligent Software Technology（北京多媒体与智能软件技术重点实验室）；Faculty of Information Technology, Beijing University of Technology（信息科技学院，北京理工大学）；University of California at Merced（加州大学默塞德分校）；Macquarie University（麦考瑞大学）
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Shanghai AI Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；Rutgers University（罗格斯大学）；Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心）
IPVTON: Image-based 3D Virtual Try-on with Image Prompt Adapter
School of Software Engineering, South China University of Technology（华南理工大学软件学院）；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Peng Cheng Laboratory（鹏城实验室）
UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval
Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）；Dalian University of Technology（大连理工大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Microsoft Research（微软研究院）
Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；School of Information Science and Engineering（信息科学与工程学院）；School of Mathematics（数学学院）；Purple Mountain Laboratories（紫金山实验室）
Mind the Trojan Horse: Image Prompt Adapter Enabling Scalable and Deceptive Jailbreaking
School of Computer Science and Engineering, Sun Yat-Sen University（中山大学计算机科学与工程学院）；Nanyang Technological University（南洋理工大学）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explor…
Urban Waterlogging Detection: A Challenging Benchmark and Large-Small Model Co-Adapter
School of Microelectronics and Communication Engineering, Chongqing University（重庆大学微电子与通信工程学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
National University of Singapore（新加坡国立大学）；Berkeley（加州大学伯克利分校）
Though vision transformers (ViTs) have exhibited impressive ability for representation learning, we empirically find that they cannot generalize well to unseen domains with previous domain generalization algorithms. In this paper, we propose a novel approach DoPrompt based on prompt learning to embed the knowledge of source domains in domain prompts for target domain prediction. Specifically, domain prompts are prepended before ViT input tokens from the corresponding source domain. Each domain prompt learns domain-specific knowledge efficiently since it is optimized only for one domain. Meanwh…
Recently, fine-tuning language models pre-trained on large text corpora have provided huge improvements on vision-and-language (V&L) tasks as well as on pure language tasks. However, fine-tuning the entire parameter set of pre-trained models becomes impractical since the model size is growing rapidly. Hence, in this paper, we introduce adapter-based parameter-efficient transfer learning techniques to V&L models such as VL-BART and VLT5. We evaluate our methods in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets:…
Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learning
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（高性能计算研究所，科技研究局）；University of Minnesota（明尼苏达大学）；Microsoft Research Asia Singapore（微软亚洲研究院新加坡分部）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（Mohamed bin Zayed人工智能大学）；Australian National University（澳大利亚国立大学）；Harbin Institute of Technology（哈尔滨工业大学）
Spectral Adapters for Segment Anything Model-based Segmentation of Colorectal Liver Metastases in Computed Tomography
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Toronto General Hospital Research Institute, University Health Network（大学健康网络多伦多总医院研究所）
Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation
Centre for AI, DS\&AI, Astrazeneca, UK（英国阿斯利康人工智能中心）；Institute for Imaging, Data and Communications (IDCOM), School of Engineering, University of Edinburgh, Edinburgh, UK（爱丁堡大学工程学院影像、数据与通信研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
