arXiv 2606.245702026-06-25cs.CV

Jolia: 用于3D CT对比学习的概念级视觉-语言对齐

Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning

Raidium；Department of Vascular and Oncological Interventional Radiology, Hôpital Européen Georges Pompidou, AP-HP（欧洲乔治·蓬皮杜医院血管与肿瘤介入放射科，AP-HP）；Faculté de Santé, Université Paris-Cité（巴黎西岱大学健康学院）；HEKA, INRIA（HEKA，法国国家信息与自动化研究所）；Imaging Department, Fondation Ophtalmologique Adolphe de Rothschild（阿道夫·罗斯柴尔德眼科基金会影像科）

Julien Khlaut, Charles Corbière, Baptiste Callard, Amaury Prat, Leo Butsanets, Antoine Saporta, Théo Danielou, Leo Machado, Korentin Le Floch, Tom Boeken, Pierr… 展开作者

Julien Khlaut, Charles Corbière, Baptiste Callard, Amaury Prat, Leo Butsanets, Antoine Saporta, Théo Danielou, Leo Machado, Korentin Le Floch, Tom Boeken, Pierre Manceron, Corentin Dancette

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出ConQuer方法，通过概念查询将报告拆分为特定概念部分，学习交叉注意力查询以匹配图像特征，实现概念级对齐，在胸部与腹部CT上训练的Jolia模型在多项任务中超越CLIP基线。

摘要：

视觉-语言对比预训练已成为3D医学基础模型的主流方法，利用临床实践中产生的大量配对扫描和报告。然而，医学图像通常涵盖数十个器官，放射学报告比典型的自然图像描述长得多，并由多个结构化部分组成。CLIP风格的预训练通过将每个模态编码为单个全局标记来压缩这种结构，存在丢失重要细节的风险。我们提出ConQuer（概念查询），一种图像-文本预训练方法，通过一组局部对齐（每个概念一个）增强CLIP的全局对齐。ConQuer将报告拆分为特定概念部分，并学习交叉注意力查询以池化匹配的图像特征，无需使用任何分割掩码或空间监督。然后对每个概念独立应用对比学习。概念可以是任何语义定位单元；这里，它们是解剖区域，每个器官或大体身体区域一个查询。作为副产品，每个查询学习聚焦于其概念的注意力图，提供内置的空间可解释性。我们使用ConQuer训练Jolia，一个在胸部与腹部CT上的3D CT基础模型。Jolia在发现分类、报告生成和跨中心迁移方面始终优于CLIP基线，并在多个公共基准上创下最新技术水平。Jolia的权重将在接收后发布。

英文摘要：

Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP's global alignment with a set of localized alignments, one per concept. ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia's weights are available at https://huggingface.co/raidium/Jolia

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。