arXiv 2503.149982026-05-08cs.CV

表格引导视觉：通过表格数据学习看见心脏

Tables Guide Vision: Learning to See the Heart through Tabular Data

School of Computation, Information and Technology, Technical University of Munich（技术大学慕尼黑计算、信息与技术学院）；Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich（生物医学成像中的机器学习研究所，海德堡慕尼黑）；School of Biomedical Engineering and Imaging Sciences, King’s College London（国王学院伦敦生物医学工程与成像科学学院）；TUM University Hospital, Technical University of Munich（技术大学慕尼黑大学医院）

Marta Hasny, Maxime Di Folco, Keno Bressem, Julia Schnabel

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于表格数据的对比学习框架，利用临床相关表格数据识别患者层面相似性，构建更有意义的配对，提升语义对齐的表示学习，并通过零样本预测验证方法有效性。

摘要：

计算机视觉中的对比学习方法通常依赖于同一图像的增强视图或多模态预训练策略来对齐配对模态。然而，这些方法往往忽略了不同实例之间的语义关系，导致在语义相似样本被当作负样本时产生假阴性。这种限制在医学影像领域尤其关键，如心脏病学，其中人口统计学和临床特征在评估疾病风险和患者结果中起关键作用。我们引入了一种表格引导的对比学习框架，利用临床相关的表格数据来识别患者层面的相似性，并构建更有意义的配对，从而在不需要跨模态联合嵌入的情况下实现语义对齐的表示学习。此外，我们还适应k-NN算法进行零样本预测，以克服单模态表示中缺乏零样本能力的不足。我们通过大量短轴心脏MRI图像和临床特征的大型队列展示了方法的强度，其中表格数据有助于更有效地区分患者亚组。在下游任务，包括微调、线性探测和心血管动脉疾病和心脏表型的零样本预测中，评估表明，结合表格数据指导的方法比仅依赖图像增强或结合图像-表格嵌入的传统方法产生了更强的视觉表示。此外，我们还通过在汽车广告数据集上评估证明了该方法可以泛化到自然图像。代码可在https://github.com/marteczkah/tables_guide_vision获取。

英文摘要：

Contrastive learning methods in computer vision typically rely on augmented views of the same image or multimodal pretraining strategies that align paired modalities. However, these approaches often overlook semantic relationships between distinct instances, leading to false negatives when semantically similar samples are treated as negatives. This limitation is especially critical in medical imaging domains such as cardiology, where demographic and clinical attributes play a critical role in assessing disease risk and patient outcomes. We introduce a tabular-guided contrastive learning framework that leverages clinically relevant tabular data to identify patient-level similarities and construct more meaningful pairs, enabling semantically aligned representation learning without requiring joint embeddings across modalities. Additionally, we adapt the k-NN algorithm for zero-shot prediction to overcome the lack of zero-shot capability in unimodal representations. We demonstrate the strength of our methods using a large cohort of short-axis cardiac MR images and clinical attributes, where tabular data helps to more effectively distinguish between patient subgroups. Evaluation on downstream tasks, including fine-tuning, linear probing, and zero-shot prediction of cardiovascular artery diseases and cardiac phenotypes, shows that incorporating tabular data guidance yields stronger visual representations than conventional methods that rely solely on image augmentation or combined image-tabular embeddings. Further, we show that our method can generalize to natural images by evaluating it on a car advertisement dataset. Code is available at https://github.com/marteczkah/tables_guide_vision.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。