arXiv 2509.152342026-06-01cs.CV

探索大语言模型编码器在胸部X光片图像-文本检索中的能力

Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays

Interdisciplinary Program in Bioengineering, Seoul National University Graduate School（生物工程跨学科项目，首尔国立大学研究生院）；Integrated Major in Innovative Medical Science, Seoul National University Graduate School（创新医学科学整合专业，首尔国立大学研究生院）；Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院第一附属医院放射科）；Seoul National University College of Medicine（首尔国立大学医学院）；Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital（首尔国立大学医学院放射科，首尔国立大学医院）；Institute of Medical and Biological Engineering, Seoul National University Medical Research Center（医学与生物工程研究所，首尔国立大学医学研究所以及）；Institute of Radiation Medicine, Seoul National University Medical Research Center（放射医学研究所，首尔国立大学医学研究所以及）

Hanbin Ko, Gihun Cho, Inhyeok Baek, Donguk Kim, Joonbeom Koo, Changi Kim, Dongheon Lee, Chang Min Park

备注：: 12 pages, 2 figures, under review

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一种领域自适应的双向大语言模型文本编码器，通过掩码标记预测和监督对比学习训练，结合参数高效的双塔对比视觉语言框架，提升胸部X光片图像与文本的对齐和检索性能。

摘要：

从配对的医学图像和临床文本中进行多模态学习是医学数据驱动信息学中的核心挑战，其中有效的跨模态对齐对于可扩展的分析和检索至关重要。在胸部放射学中，视觉语言预训练受到异质性放射学报告的制约，这些报告包含缩写、仅印象笔记和机构特定的写作风格。与通用领域不同，当报告风格差异显著时，简单聚合大量噪声报告可能会使多模态学习停滞甚至退化。我们提出了一种针对胸部放射学报告的领域自适应双向大语言模型文本编码器，通过在风格多样但临床等效的报告变体上进行掩码标记预测和监督对比学习训练，以生成鲁棒、可泛化的文本嵌入。然后，我们使用参数高效适配将该编码器集成到双塔对比视觉语言框架中，以改善图像-文本对齐。在来自公共数据集和去标识化医院队列的160万对配对研究上，所提出的模型提高了双向检索准确性和外部泛化能力，在MIMIC-CXR上达到0.308的GREEN分数，在Open-I上达到0.618，同时减少了在训练中添加富含缩写、仅印象的医院报告时观察到的退化。

英文摘要：

Multimodal learning from paired medical images and clinical text is a central challenge in medical data-driven informatics, where effective cross-modal alignment is critical for scalable analysis and retrieval. In chest radiography, vision-language pretraining is constrained by heterogeneous radiology reports that contain abbreviations, impression-only notes, and institution-specific writing styles. Unlike general-domain settings, naively aggregating large collections of noisy reports can plateau or even degrade multimodal learning when reporting styles differ substantially. We propose a domain-adapted bidirectional large language model text encoder for chest radiograph reports, trained with masked token prediction and supervised contrastive learning on stylistically diverse but clinically equivalent report variants to produce robust, generalizable text embeddings. We then integrate this encoder into a dual-tower contrastive vision-language framework using parameter-efficient adaptation to improve image-text alignment. Across 1.6 million paired studies from public datasets and a de-identified hospital cohort, the proposed models improve bidirectional retrieval accuracy and external generalization, achieving GREEN scores of 0.308 on MIMIC-CXR and 0.618 on Open-I, while reducing the degradation observed when abbreviation-rich, impression-only hospital reports are added to training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。