arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.07355cs.CVcs.LG
收藏

IMITATE:临床先验引导的分层视觉语言预训练

IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training

Imperial College London(帝国理工学院);École des Ponts(巴黎路桥学院);EDF R&D(法国电力公司研发中心);Royal Brompton and Harefield Hospital(皇家布朗普顿和 Harefield 医院);Tongji University(同济大学)

Che Liu, Sibo Cheng, Miaojing Shi, Anand Shah, Wenjia Bai, Rossella Arcucci

AI总结:

针对现有医学VLP忽略临床报告分层结构的问题,提出临床先验引导的分层视觉语言预训练框架IMITATE,通过多级视觉特征与分层文本对齐及临床信息对比损失提升性能,在多个下游任务上优于基线方法。

摘要:

在医学视觉语言预训练(VLP)领域,大量研究致力于从临床报告及对应的医学图像中提取文本和图像特征。然而,现有大多数方法可能忽略了利用临床报告固有分层结构的价值——这类报告通常分为描述内容的“影像表现(findings)”和结论性观察的“影像印象(impressions)”两部分。当前的医学VLP方法往往将报告简化为统一实体或碎片化token,并未利用这种丰富的结构化格式。本研究提出一种名为IMITATE的新型临床先验引导VLP框架,通过分层视觉语言对齐从医学报告中学习结构信息。该框架从胸部X射线(CXR)图像中提取多级视觉特征,并将这些特征分别与分层医学报告中编码的描述性文本和结论性文本分别对齐。此外,本文提出一种新的临床信息对比损失用于跨模态学习,该损失在构建对比学习中的样本相关性时纳入了临床先验知识。所提出的IMITATE模型在6个不同数据集、涵盖5个医学影像下游任务上的表现优于基线VLP方法。综合实验结果凸显了将医学报告的分层结构融入视觉语言对齐的优势。本文相关代码可在https://github.com/cheliu-computation/IMITATE-TMI2024获取。

英文摘要:

In the field of medical Vision-Language Pre-training (VLP), significant efforts have been devoted to deriving text and image features from both clinical reports and associated medical images. However, most existing methods may have overlooked the opportunity in leveraging the inherent hierarchical structure of clinical reports, which are generally split into `findings' for descriptive content and `impressions' for conclusive observation. Instead of utilizing this rich, structured format, current medical VLP approaches often simplify the report into either a unified entity or fragmented tokens. In this work, we propose a novel clinical prior guided VLP framework named IMITATE to learn the structure information from medical reports with hierarchical vision-language alignment. The framework derives multi-level visual features from the chest X-ray (CXR) images and separately aligns these features with the descriptive and the conclusive text encoded in the hierarchical medical report. Furthermore, a new clinical-informed contrastive loss is introduced for cross-modal learning, which accounts for clinical prior knowledge in formulating sample correlations in contrastive learning. The proposed model, IMITATE, outperforms baseline VLP methods across six different datasets, spanning five medical imaging downstream tasks. Comprehensive experimental results highlight the advantages of integrating the hierarchical structure of medical reports for vision-language alignment. The code related to this paper is available at https://github.com/cheliu-computation/IMITATE-TMI2024.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑