arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00650cs.CV
收藏

Segment Anything Model 的深度指令微调

Deep Instruction Tuning for Segment Anything Model

Xiamen University(厦门大学);The Hong Kong University of Science and Technology(香港科技大学)

Xiaorui Huang, Gen Luo, Chaoyang Zhu, Bo Tong, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji

AI总结:

本文提出两种深度指令微调方法(端到端和逐层),将SAM的图像编码器转变为独立视觉语言学习器,大幅提升了其在指代图像分割任务上的性能并达到SOTA。

摘要:

最近,Segment Anything Model (SAM) 已成为多媒体和计算机视觉领域的研究热点,它在各种(无)条件图像分割任务中展现出强大且通用的能力。尽管 SAM 能支持多种类型的分割提示,但我们注意到,与点引导和框引导分割相比,它在文本指令任务上表现差得多,例如,指代图像分割 (RIS)。在本文中,我们认为深度文本指令微调是缓解其默认轻量级掩码解码器中浅层融合方案所导致缺陷的关键。为了解决这个问题,我们为 SAM 提出了两种简单而有效的深度指令微调 (DIT) 方法,一种是端到端的,另一种是逐层的。只需极少的修改,DIT 就能直接将 SAM 的图像编码器转变为独立的视觉语言学习器,而无需构建另一个深层融合分支,从而最大化其卓越分割能力的收益。在三个高度竞争的 RIS 基准数据集上的广泛实验表明,简单的端到端 DIT 可以大幅提升 SAM 的性能,而逐层 DIT 能以更少的数据和训练开销进一步提升至最先进的性能。我们的代码已发布于:https://github.com/wysnzzzz/DIT。

英文摘要:

Recently, Segment Anything Model (SAM) has become a research hotspot in the fields of multimedia and computer vision, which exhibits powerful yet versatile capabilities on various (un) conditional image segmentation tasks. Although SAM can support different types of segmentation prompts, we note that, compared to point- and box-guided segmentations, it performs much worse on text-instructed tasks, e.g., referring image segmentation (RIS). In this paper, we argue that deep text instruction tuning is key to mitigate such shortcoming caused by the shallow fusion scheme in its default light-weight mask decoder. To address this issue, we propose two simple yet effective deep instruction tuning (DIT) methods for SAM, one is end-to-end and the other is layer-wise. With minimal modifications, DITs can directly transform the image encoder of SAM as a stand-alone vision-language learner in contrast to building another deep fusion branch, maximizing the benefit of its superior segmentation capability. Extensive experiments on three highly competitive benchmark datasets of RIS show that a simple end-to-end DIT can improve SAM by a large margin, while the layer-wise DIT can further boost the performance to state-of-the-art with much less data and training expenditures. Our code is released at: https://github.com/wysnzzzz/DIT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑