arXiv 2402.157592024-10-14cs.CVcs.AI

TV-SAM：使用GPT-4生成的描述性提示且无需人工标注以提升多模态医学图像的零样本分割性能

TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation

Sichuan University（四川大学）；Case Western Reserve University（凯斯西储大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Urgench State University（乌尔根奇国立大学）；Gansu Agricultural University（甘肃农业大学）

Zekun Jiang, Dongjie Cheng, Ziyuan Qin, Jun Gao, Qicheng Lao, Abdullaev Bakhrom Ismoilovich, Urazboev Gayrat, Yuldashov Elyorbek, Bekchanov Habibullo, Defu Tang… 展开作者

Zekun Jiang, Dongjie Cheng, Ziyuan Qin, Jun Gao, Qicheng Lao, Abdullaev Bakhrom Ismoilovich, Urazboev Gayrat, Yuldashov Elyorbek, Bekchanov Habibullo, Defu Tang, LinJing Wei, Kang Li, Le Zhang

备注：: 13 pages, 5 figures, 4 tables, accepted by BDMA Journal

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出无需人工标注的零样本分割算法TV-SAM，通过集成GPT-4、GLIP和SAM自主生成文本与视觉提示，在7个数据集8种模态上表现优异，突显基础模型在专业领域的应用价值。

摘要：

本研究提出了一种名为文本-视觉提示分割一切模型（TV-SAM）的新型多模态医学图像零样本分割算法，无需任何人工标注。TV-SAM结合并集成了大语言模型GPT-4、视觉语言模型GLIP和SAM，能够从医学图像中自主生成描述性文本提示和视觉边界框提示，从而增强SAM的零样本分割能力。研究在包含8种成像模态的7个公开数据集上进行了全面评估，证明TV-SAM无需额外训练即可有效分割跨各种模态的未见目标。TV-SAM显著优于SAM AUTO和GSAM，其性能紧密匹配使用金标准边界框提示的SAM BBOX，并在ISIC和WBC等特定数据集上超越了现有最先进方法。研究表明，TV-SAM是一种有效的多模态医学图像零样本分割算法，突显了GPT-4对零样本分割的显著贡献。通过集成GPT-4、GLIP和SAM等基础模型，能够增强解决专业领域复杂问题的能力。

英文摘要：

This study presents a novel multimodal medical image zero-shot segmentation algorithm named the text-visual-prompt segment anything model (TV-SAM) without any manual annotations. The TV-SAM incorporates and integrates the large language model GPT-4, the vision language model GLIP, and the SAM to autonomously generate descriptive text prompts and visual bounding box prompts from medical images, thereby enhancing the SAM's capability for zero-shot segmentation. Comprehensive evaluations are implemented on seven public datasets encompassing eight imaging modalities to demonstrate that TV-SAM can effectively segment unseen targets across various modalities without additional training. TV-SAM significantly outperforms SAM AUTO and GSAM, closely matching the performance of SAM BBOX with gold standard bounding box prompts and surpasses the state-of-the-art methods on specific datasets such as ISIC and WBC. The study indicates that TV-SAM serves as an effective multimodal medical image zero-shot segmentation algorithm, highlighting the significant contribution of GPT-4 to zero-shot segmentation. By integrating foundational models such as GPT-4, GLIP, and SAM, the ability to address complex problems in specialized domains can be enhanced.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。