arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.09294cs.CVcs.CL
收藏

解剖结构引导的医学视觉-语言预训练

Anatomical Structure-Guided Medical Vision-Language Pre-training

Fudan University(复旦大学);Tongji University(同济大学);The Hong Kong Polytechnic University(香港理工大学);Research Institute for Smart Ageing(智能老龄化研究院)

Qingqiu Li, Xiaohan Yan, Jilan Xu, Runtian Yuan, Yuejie Zhang, Rui Feng, Quanli Shen, Xiaobo Zhang, Shujun Wang

AI总结:

针对医学视觉-语言预训练中局部对齐缺乏可解释性及表示学习不足的问题,提出解剖结构引导(ASG)框架,通过解析报告为三元组并利用各元素监督,在五个公共基准上取得最优性能。

摘要:

通过视觉-语言预训练学习医学视觉表示已取得显著进展。尽管性能令人期待,但仍面临挑战,即局部对齐缺乏可解释性和临床相关性,以及图像-报告对的内部和外部表示学习不足。为解决这些问题,我们提出了一种解剖结构引导(ASG)框架。具体而言,我们将原始报告解析为三元组<解剖区域,发现,存在性>,并充分利用每个元素作为监督以增强表示学习。对于解剖区域,我们与放射科医生合作设计了一种自动的解剖区域-句子对齐范式,将其视为最小语义单元以探索细粒度的局部对齐。对于发现和存在性,我们将其视为图像标签,应用图像-标签识别解码器将图像特征与每个样本中各自的标签关联起来,并为对比学习构建软标签,以改善不同图像-报告对的语义关联。我们在两个下游任务上评估了所提出的ASG框架,包括五个公共基准。实验结果表明,我们的方法优于最先进的方法。

英文摘要:

Learning medical visual representations through vision-language pre-training has reached remarkable progress. Despite the promising performance, it still faces challenges, i.e., local alignment lacks interpretability and clinical relevance, and the insufficient internal and external representation learning of image-report pairs. To address these issues, we propose an Anatomical Structure-Guided (ASG) framework. Specifically, we parse raw reports into triplets <anatomical region, finding, existence>, and fully utilize each element as supervision to enhance representation learning. For anatomical region, we design an automatic anatomical region-sentence alignment paradigm in collaboration with radiologists, considering them as the minimum semantic units to explore fine-grained local alignment. For finding and existence, we regard them as image tags, applying an image-tag recognition decoder to associate image features with their respective tags within each sample and constructing soft labels for contrastive learning to improve the semantic association of different image-report pairs. We evaluate the proposed ASG framework on two downstream tasks, including five public benchmarks. Experimental results demonstrate that our method outperforms the state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑