arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.05239cs.CVcs.CL
收藏

TuneVLSeg:面向视觉语言分割模型的提示调优基准

TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models

Nepal Applied Mathematics and Informatics Institute for research (NAAMII)(尼泊尔应用数学与信息学研究所(NAAMII))

Rabin Adhikari, Safal Thapaliya, Manish Dhakal, Bishesh Khanal

AI总结:

该研究提出开源基准框架TuneVLSeg,整合多种提示调优技术到视觉语言分割模型,在多类医学及自然域分割数据集上测试,发现视觉提示调优性价比突出,推进了域特定分割相关技术的应用。

摘要:

视觉语言模型(Vision-Language Models, VLMs)在视觉任务中展现出优异性能,但将其适配到新域通常需要成本高昂的微调。提示调优技术(包括文本、视觉和多模态提示)通过可学习提示提供了高效替代方案,但它们在视觉语言分割模型(Vision-Language Segmentation Models, VLSMs)上的应用,以及在显著域偏移下的评估仍未被探索。本工作提出了一个开源基准框架TuneVLSeg,将多种单模态和多模态提示调优技术整合到VLSMs中,使提示调优可用于任意类别数的下游分割数据集。TuneVLSeg涵盖了应用于2种VLSMs的6种提示调优策略(采用不同提示深度),共计8种不同组合。我们在8个多样化的医学数据集(包括3个放射学数据集:乳腺肿瘤、超声心动图、胸部X光病变,以及5个非放射学数据集:息肉、溃疡、皮肤癌)和2个自然域分割数据集上测试了多种提示调优方法。研究发现,在从自然域图像到医学数据的显著域偏移下,文本提示调优表现不佳。此外,视觉提示调优的超参数少于多模态提示调优,却通常能达到与多模态方法相当的性能,是值得优先尝试的方案。本工作推进了对不同提示调优技术在鲁棒的域特定分割任务中的理解与适用性。源代码可在https://github.com/naamiinepal/tunevlseg 获取。

英文摘要:

Vision-Language Models (VLMs) have shown impressive performance in vision tasks, but adapting them to new domains often requires expensive fine-tuning. Prompt tuning techniques, including textual, visual, and multimodal prompting, offer efficient alternatives by leveraging learnable prompts. However, their application to Vision-Language Segmentation Models (VLSMs) and evaluation under significant domain shifts remain unexplored. This work presents an open-source benchmarking framework, TuneVLSeg, to integrate various unimodal and multimodal prompt tuning techniques into VLSMs, making prompt tuning usable for downstream segmentation datasets with any number of classes. TuneVLSeg includes $6$ prompt tuning strategies on various prompt depths used in $2$ VLSMs totaling of $8$ different combinations. We test various prompt tuning on $8$ diverse medical datasets, including $3$ radiology datasets (breast tumor, echocardiograph, chest X-ray pathologies) and $5$ non-radiology datasets (polyp, ulcer, skin cancer), and two natural domain segmentation datasets. Our study found that textual prompt tuning struggles under significant domain shifts, from natural-domain images to medical data. Furthermore, visual prompt tuning, with fewer hyperparameters than multimodal prompt tuning, often achieves performance competitive to multimodal approaches, making it a valuable first attempt. Our work advances the understanding and applicability of different prompt-tuning techniques for robust domain-specific segmentation. The source code is available at https://github.com/naamiinepal/tunevlseg.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑