arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.01522cs.CVcs.LG
收藏

G2D:通过视觉-语言预训练实现从全局到密集的放射影像表征学习

G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

Imperial College London(帝国理工学院);University of Oxford(牛津大学);École des Ponts(巴黎桥路学校);EDF R&D(法国电力公司研发中心);Royal Brompton and Harefield Hospitals(皇家布朗普顿和 Harefield 医院)

Che Liu, Cheng Ouyang, Sibo Cheng, Anand Shah, Wenjia Bai, Rossella Arcucci

AI总结:

本文提出G2D医学视觉-语言预训练框架,通过并行的无参数伪分割任务学习密集语义表征,在6项任务、25种疾病上表现优越,语义分割中仅用1%数据微调即超越使用100%数据的同类模型。

摘要:

近年来,医学视觉-语言预训练(VLP)在从医学图像及其配对放射学报告中学习全局视觉表征方面取得了实质性进展。然而,现实世界中的医学影像任务通常需要更细粒度的视觉特征。这些任务包括视觉定位任务(例如语义分割、目标检测)以及视觉指代定位任务。但是,当前的医学VLP方法在学习这些细粒度特征方面面临挑战,因为它们主要关注图像块与单个文本token之间的暴力对齐以进行局部视觉特征学习,这对于下游密集预测任务而言并非最优。在本研究中,我们提出了一个名为Global to Dense level representation learning(G2D)的新型VLP框架,与现有医学VLP方法相比,该框架显著提升了所学特征的粒度并实现了更准确的指代定位。具体而言,G2D通过与全局视觉-语言对齐并行的伪分割任务,学习密集且具有语义根基的图像表征。值得注意的是,生成伪分割目标不会产生额外的可训练参数:这些目标是在VLP过程中通过无参数处理器即时获得的。G2D在6项医学影像任务和25种疾病上均取得了优越性能,尤其是在需要细粒度、语义根基图像特征的语义分割任务中。在该任务上,即使仅使用1%的训练数据进行微调,而对比模型使用100%的训练数据,G2D仍优于这些同类模型。代码可在https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main获取。

英文摘要:

Recently, medical vision-language pre-training (VLP) has reached substantial progress to learn global visual representation from medical images and their paired radiology reports. However, medical imaging tasks in real world usually require finer granularity in visual features. These tasks include visual localization tasks (e.g., semantic segmentation, object detection) and visual grounding task. Yet, current medical VLP methods face challenges in learning these fine-grained features, as they primarily focus on brute-force alignment between image patches and individual text tokens for local visual feature learning, which is suboptimal for downstream dense prediction tasks. In this work, we propose a new VLP framework, named \textbf{G}lobal to \textbf{D}ense level representation learning (G2D) that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches. In particular, G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. Notably, generating pseudo segmentation targets does not incur extra trainable parameters: they are obtained on the fly during VLP with a parameter-free processor. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation, which necessitates fine-grained, semantically-grounded image features. In this task, G2D surpasses peer models even when fine-tuned with just 1\% of the training data, compared to the 100\% used by these models. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑