本研究提出利用冻结权重的 Latent Diffusion Model 进行零样本医学短语定位,通过特征选择与后处理策略在胸部X光基准上取得与SOTA相当甚至更优的性能。
摘要:
在给定的医学扫描图像中精确定位病灶区域是一个重要的成像问题,传统上需要大量的边界框真实标注才能准确解决。然而,存在其他可能较弱的监督形式,例如伴随的自由文本报告,这些是很容易获取的。在文本引导下执行定位的任务通常被称为短语定位。在本研究中,我们使用公开可用的基础模型,即 Latent Diffusion Model,来执行这一具有挑战性的任务。这一选择是基于以下事实:Latent Diffusion Model 尽管本质上是生成式的,但包含交叉注意力机制,能够隐式地对齐视觉和文本特征,从而产生适合当前任务的中间表示。此外,我们旨在以零样本方式执行此任务,即在目标任务上没有任何训练,这意味着模型的权重保持冻结。为此,我们设计了策略来选择特征,并通过后处理对其进行细化,且无需额外的可学习参数。我们将提出的方法与通过对比学习在联合嵌入空间中显式强制执行图像-文本对齐的最先进方法进行了比较。在一个流行的胸部X光基准上的结果表明,我们的方法在不同类型的病理上与SOTA具有竞争力,甚至在两项指标(平均IoU和AUC-ROC)上平均优于它们。源代码将在被接受后发布于 https://github.com/vios-s。
英文摘要:
Localizing the exact pathological regions in a given medical scan is an important imaging problem that traditionally requires a large amount of bounding box ground truth annotations to be accurately solved. However, there exist alternative, potentially weaker, forms of supervision, such as accompanying free-text reports, which are readily available. The task of performing localization with textual guidance is commonly referred to as phrase grounding. In this work, we use a publicly available Foundation Model, namely the Latent Diffusion Model, to perform this challenging task. This choice is supported by the fact that the Latent Diffusion Model, despite being generative in nature, contains cross-attention mechanisms that implicitly align visual and textual features, thus leading to intermediate representations that are suitable for the task at hand. In addition, we aim to perform this task in a zero-shot manner, i.e., without any training on the target task, meaning that the model's weights remain frozen. To this end, we devise strategies to select features and also refine them via post-processing without extra learnable parameters. We compare our proposed method with state-of-the-art approaches which explicitly enforce image-text alignment in a joint embedding space via contrastive learning. Results on a popular chest X-ray benchmark indicate that our method is competitive with SOTA on different types of pathology, and even outperforms them on average in terms of two metrics (mean IoU and AUC-ROC). Source code will be released upon acceptance at https://github.com/vios-s.
Detector-Free Weakly Supervised Grounding by Separation
IBM Research(IBM研究院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);Tel-Aviv University(特拉维夫大学);Weizmann Institute of Science(魏茨曼科学研究所);Boston University(波士顿大学);Technion(以色列理工学院)
Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Software, Beihang University(北京航空航天大学软件学院);Meituan(美团);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程学院)
A Comparison of Object Detection and Phrase Grounding Models in Chest X-ray Abnormality Localization using Eye-tracking Data
Scientific Computing and Imaging Institute, University of Utah(犹他大学科学计算与成像研究所);Kahlert School of Computing, University of Utah(犹他大学卡勒特计算学院);University of Utah(犹他大学)
Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
The Chinese University of Hong Kong(香港中文大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tencent AI Lab(腾讯AI实验室);University of Rochester(罗切斯特大学)
Contrastive Learning for Weakly Supervised Phrase Grounding
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达);Bar Ilan University(巴伊兰大学)
短语基础,将图像区域与字幕词相关联的问题,是视觉语言任务的重要组成部分。我们表明,短语接地可以通过优化字区域的注意,以最大限度地提高图像和字幕词之间的互信息的下限。给定成对的图像和字幕,我们最大限度地提高了注意力加权区域和相应字幕中的单词的兼容性,相比于非对应的图像和字幕对。其核心思想是通过语言模型指导的词替换来构造有效的负面字幕。与来自训练数据的随机采样负数相比,使用我们的负数进行训练会在准确性方面产生$\sim10\%$绝对增益。我们在COCO-Captions上训练的弱监督短语接地模型显示了$5.7\%$的健康增益,以在Flickr 30 K Entities基准上实现$76.7\%$准确性。
Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment
Tianjin University(天津大学);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Singapore General Hospital(新加坡中央医院);Saw Swee Hock School of Public Health, National University of Singapore National University Health System(新加坡国立大学国立大学卫生系统苏瑞福公共卫生学院)
Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding
College of Computer Science, Sichuan University(四川大学计算机学院);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Department of Radiology, West China Hospital, Sichuan University(四川大学华西医院放射科);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学院);Department of Mathematics, Sichuan University(四川大学数学系);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-Sen University(中山大学深圳校区网络科学与技术学院);Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore and the Singapore Eye Research Institute, Singapore National Eye Centre(新加坡国立大学 Yong Loo Lin 医学院眼科系及新加坡眼科研所、新加坡国家眼科中心)
Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Meituan(美团);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院)
PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding
Beihang University(北京航空航天大学);Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学);Meituan(美团)
Phrase Grounding-based Style Transfer for Single-Domain Generalized Object Detection
School of Computer, National University of Defense Technology, Changsha, China(国防科技大学计算机学院,中国长沙);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China(中山大学深圳校区计算机科学与技术学院,中国深圳);Hong Kong Polytechnic University, Department of Computing, Hong Kong(香港理工大学计算学院,香港);College of Computer Science and Electronic Engineering, Hunan University, Changsha, China(湖南大学计算机科学与电子工程学院,中国长沙)
PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Inria, École normale supérieure, CNRS, PSL Research University(法国国家信息与自动化研究所,巴黎高等师范学院,法国国家科学研究中心,巴黎文理研究大学);Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague(捷克信息学、机器人与控制论研究所,布拉格捷克理工大学)
Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Yong Yoo Lin School of Medicine, National University of Singapore(新加坡国立大学杨潞龄医学院);Institute of High-Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所)