arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.12920cs.CVcs.LG
收藏

使用现成扩散模型的零样本文本短语定位

Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models

School of Engineering, University of Edinburgh(爱丁堡大学工程学院);Canon Medical Research Europe Ltd.(佳能医疗欧洲研究有限公司)

Konstantinos Vilouras, Pedro Sanchez, Alison Q. O'Neil, Sotirios A. Tsaftaris

AI总结:

本研究提出利用冻结权重的 Latent Diffusion Model 进行零样本医学短语定位,通过特征选择与后处理策略在胸部X光基准上取得与SOTA相当甚至更优的性能。

摘要:

在给定的医学扫描图像中精确定位病灶区域是一个重要的成像问题,传统上需要大量的边界框真实标注才能准确解决。然而,存在其他可能较弱的监督形式,例如伴随的自由文本报告,这些是很容易获取的。在文本引导下执行定位的任务通常被称为短语定位。在本研究中,我们使用公开可用的基础模型,即 Latent Diffusion Model,来执行这一具有挑战性的任务。这一选择是基于以下事实:Latent Diffusion Model 尽管本质上是生成式的,但包含交叉注意力机制,能够隐式地对齐视觉和文本特征,从而产生适合当前任务的中间表示。此外,我们旨在以零样本方式执行此任务,即在目标任务上没有任何训练,这意味着模型的权重保持冻结。为此,我们设计了策略来选择特征,并通过后处理对其进行细化,且无需额外的可学习参数。我们将提出的方法与通过对比学习在联合嵌入空间中显式强制执行图像-文本对齐的最先进方法进行了比较。在一个流行的胸部X光基准上的结果表明,我们的方法在不同类型的病理上与SOTA具有竞争力,甚至在两项指标(平均IoU和AUC-ROC)上平均优于它们。源代码将在被接受后发布于 https://github.com/vios-s。

英文摘要:

Localizing the exact pathological regions in a given medical scan is an important imaging problem that traditionally requires a large amount of bounding box ground truth annotations to be accurately solved. However, there exist alternative, potentially weaker, forms of supervision, such as accompanying free-text reports, which are readily available. The task of performing localization with textual guidance is commonly referred to as phrase grounding. In this work, we use a publicly available Foundation Model, namely the Latent Diffusion Model, to perform this challenging task. This choice is supported by the fact that the Latent Diffusion Model, despite being generative in nature, contains cross-attention mechanisms that implicitly align visual and textual features, thus leading to intermediate representations that are suitable for the task at hand. In addition, we aim to perform this task in a zero-shot manner, i.e., without any training on the target task, meaning that the model's weights remain frozen. To this end, we devise strategies to select features and also refine them via post-processing without extra learnable parameters. We compare our proposed method with state-of-the-art approaches which explicitly enforce image-text alignment in a joint embedding space via contrastive learning. Results on a popular chest X-ray benchmark indicate that our method is competitive with SOTA on different types of pathology, and even outperforms them on average in terms of two metrics (mean IoU and AUC-ROC). Source code will be released upon acceptance at https://github.com/vios-s.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑