ViKL：一种通过聚合视觉-知识-语言特征的多模态乳腺X线摄影解释框架
ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features
AI总结：
提出ViKL框架，通过三重对比学习融合视觉、知识和语言特征进行无监督预训练，并发布MVKL多模态乳腺X线摄影数据集，显著提升了病理分类、泛化能力及表示质量。
摘要：
乳腺X线摄影是乳腺癌诊断的主要影像学工具。尽管在应用深度学习解释乳腺X线摄影图像方面取得了显著进展，但主要依赖视觉特征的方法通常难以在不同数据集间泛化。我们假设，在放射学实践中整合额外模态，特别是报告的语言特征和体现放射学见解的表征特征，能够提供更强大、更具可解释性和泛化性的表示。本文发布了MVKL，这是首个包含多视图图像、详细表征和报告的多模态乳腺X线摄影数据集。基于此数据集，我们聚焦于无监督预训练这一具有挑战性的任务，并提出ViKL，一个融合视觉、知识和语言特征的创新框架。该框架仅依赖配对信息，无需通常难以获取的病理标签。ViKL采用三重对比学习方法，将语言和基于知识的见解与视觉数据融合，实现跨模态和模态内特征增强。研究得出重要发现：1）将报告和表征与无监督视觉预训练相结合，ViKL显著提升了病理分类能力并促进多模态交互。2）表征可引入新的困难负样本选择机制。3）多模态特征在不同数据集间展现出可迁移性。4）多模态预训练方法抑制了错误校准并构建了高质量表示空间。MVKL数据集和ViKL代码已在https://github.com/wxwxwwxxx/ViKL公开发布，以支持广泛的未来研究。
英文摘要：
Mammography is the primary imaging tool for breast cancer diagnosis. Despite significant strides in applying deep learning to interpret mammography images, efforts that focus predominantly on visual features often struggle with generalization across datasets. We hypothesize that integrating additional modalities in the radiology practice, notably the linguistic features of reports and manifestation features embodying radiological insights, offers a more powerful, interpretable and generalizable representation. In this paper, we announce MVKL, the first multimodal mammography dataset encompassing multi-view images, detailed manifestations and reports. Based on this dataset, we focus on the challanging task of unsupervised pretraining and propose ViKL, a innovative framework that synergizes Visual, Knowledge, and Linguistic features. This framework relies solely on pairing information without the necessity for pathology labels, which are often challanging to acquire. ViKL employs a triple contrastive learning approach to merge linguistic and knowledge-based insights with visual data, enabling both inter-modality and intra-modality feature enhancement. Our research yields significant findings: 1) Integrating reports and manifestations with unsupervised visual pretraining, ViKL substantially enhances the pathological classification and fosters multimodal interactions. 2) Manifestations can introduce a novel hard negative sample selection mechanism. 3) The multimodal features demonstrate transferability across different datasets. 4) The multimodal pretraining approach curbs miscalibrations and crafts a high-quality representation space. The MVKL dataset and ViKL code are publicly available at https://github.com/wxwxwwxxx/ViKL to support a broad spectrum of future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
