arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.08059cs.CVcs.AIcs.CLcs.LG
收藏

FluoroSAM:一种用于灵活X射线图像分割的语言可提示基础模型

FluoroSAM: A Language-promptable Foundation Model for Flexible X-ray Image Segmentation

Johns Hopkins University(约翰霍普金斯大学);University of Arkansas(阿肯色大学);Johns Hopkins Hospital(约翰霍普金斯医院)

Benjamin D. Killeen, Liam J. Wang, Blanca Inigo, Han Zhang, Mehran Armand, Russell H. Taylor, Greg Osgood, Mathias Unberath

AI总结:

本文提出FluoroSAM,一种基于Segment Anything Model的语言可提示基础模型,通过在3M张合成X射线图像上从零训练并结合文本嵌入的向量量化技术,实现了基于自然语言提示的解剖结构和工具分割,为X射线图像分析中的人机交互提供了关键支持。

摘要:

语言可提示的X射线图像分割将为诊断和介入精准医学中的人机协同工作流程提供更大的灵活性。先前的努力贡献了能够解决狭窄范围内问题的任务特定模型,但扩展到更广泛的用途需要额外的数据、标注和训练时间。最近,语言对齐的基础模型(LFMs)——在海量高度可变的图像和文本数据上训练的机器学习模型,从而实现广泛的适用性——已成为自动化图像分析的有前途的工具。现有的医学图像分析基础模型集中在有大型、丰富标注数据集可用的场景和模态上。然而,X射线成像模态具有高度可变的图像外观和应用,从诊断性胸部X射线到介入透视,数据的可用性各不相同。为了为任意医学X射线图像的全面和语言对齐分析的LFM铺平道路,我们引入了FluoroSAM,这是Segment Anything Model的语言可提示变体,从零开始在来自各种人体解剖结构、成像几何和视角的3M张合成X射线图像上进行训练。这些图像包括128种器官类型和464种工具的伪真实掩模及相关文本描述。由于在训练过程中新颖地结合了文本嵌入的向量量化(VQ),FluoroSAM能够基于自然语言提示分割无数解剖结构和工具。我们在真实X射线图像上定量展示了FluoroSAM的性能,并在几个应用中展示了FluoroSAM如何成为X射线图像获取和分析环境中丰富人机交互的关键使能器。代码可在https://github.com/arcadelab/fluorosam获取。

英文摘要:

Language promptable X-ray image segmentation would enable greater flexibility for human-in-the-loop workflows in diagnostic and interventional precision medicine. Prior efforts have contributed task-specific models capable of solving problems within a narrow scope, but expanding to broader use requires additional data, annotations, and training time. Recently, language-aligned foundation models (LFMs) -- machine learning models trained on large amounts of highly variable image and text data thus enabling broad applicability -- have emerged as promising tools for automated image analysis. Existing foundation models for medical image analysis focus on scenarios and modalities where large, richly annotated datasets are available. However, the X-ray imaging modality features highly variable image appearance and applications, from diagnostic chest X-rays to interventional fluoroscopy, with varying availability of data. To pave the way toward an LFM for comprehensive and language-aligned analysis of arbitrary medical X-ray images, we introduce FluoroSAM, a language-promptable variant of the Segment Anything Model, trained from scratch on 3M synthetic X-ray images from a wide variety of human anatomies, imaging geometries, and viewing angles. These include pseudo-ground truth masks for 128 organ types and 464 tools with associated text descriptions. FluoroSAM is capable of segmenting myriad anatomical structures and tools based on natural language prompts, thanks to the novel incorporation of vector quantization (VQ) of text embeddings in the training process. We demonstrate FluoroSAM's performance quantitatively on real X-ray images and showcase on several applications how FluoroSAM is a key enabler for rich human-machine interaction in the X-ray image acquisition and analysis context. Code is available at https://github.com/arcadelab/fluorosam.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑