arXiv 2405.061962024-06-27cs.CVcs.AIcs.CLcs.LG

VLSM-Adapter：使用轻量级模块高效微调视觉-语言分割模型

VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks

Nepal Applied Mathematics and Informatics Institute (NAAMII)（尼泊尔应用数学与信息学研究所（NAAMII））

Manish Dhakal, Rabin Adhikari, Safal Thapaliya, Bishesh Khanal

备注：: Accepted at MICCAI 2024, the 27th International Conference on Medical Image Computing and Computer Assisted Intervention

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出VLSM-Adapter，通过在冻结的预训练模型上仅训练轻量级模块，以仅300万参数高效微调医学图像视觉-语言分割模型，性能超越SOTA并媲美端到端微调。

摘要：

使用大规模开放域图文对训练的基础视觉-语言模型（VLMs）最近被适配开发为视觉-语言分割模型（VLSMs），允许在推理时提供文本提示来指导图像分割。如果能为医学图像构建稳健且强大的VLSMs，将能在许多临床任务中辅助医疗专业人员，否则他们必须花费大量时间来描绘感兴趣的目标结构。由于标注的医学图像数据集较少，医学图像的VLSMs依赖于在开放域自然图像数据集上预训练的基础VLM或VLSM进行微调；这种微调过程消耗资源且成本高昂，因为通常需要更新全部或很大一部分预训练参数。最近，VLMs中提出了名为adapters的轻量级模块，它们保持预训练模型冻结，仅在微调时训练adapters，从而大幅减少所需的计算资源。我们引入了一种新型adapter，即VLSM-Adapter，能够使用transformer编码器微调预训练的视觉-语言分割模型。我们在广泛使用的基于CLIP的分割模型上的实验表明，仅需300万个可训练参数，VLSM-Adapter就超越了现有技术，并且可媲美上限的端到端微调。源代码可在以下地址获取：https://github.com/naamiinepal/vlsm-adapter。

英文摘要：

Foundation Vision-Language Models (VLMs) trained using large-scale open-domain images and text pairs have recently been adapted to develop Vision-Language Segmentation Models (VLSMs) that allow providing text prompts during inference to guide image segmentation. If robust and powerful VLSMs can be built for medical images, it could aid medical professionals in many clinical tasks where they must spend substantial time delineating the target structure of interest. VLSMs for medical images resort to fine-tuning base VLM or VLSM pretrained on open-domain natural image datasets due to fewer annotated medical image datasets; this fine-tuning is resource-consuming and expensive as it usually requires updating all or a significant fraction of the pretrained parameters. Recently, lightweight blocks called adapters have been proposed in VLMs that keep the pretrained model frozen and only train adapters during fine-tuning, substantially reducing the computing resources required. We introduce a novel adapter, VLSM-Adapter, that can fine-tune pretrained vision-language segmentation models using transformer encoders. Our experiments in widely used CLIP-based segmentation models show that with only 3 million trainable parameters, the VLSM-Adapter outperforms state-of-the-art and is comparable to the upper bound end-to-end fine-tuning. The source code is available at: https://github.com/naamiinepal/vlsm-adapter.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。