单模态医学文本与视觉表示的语义对齐
Semantic Alignment of Unimodal Medical Text and Vision Representations
AI总结：
本文提出利用锚点估计语义对齐变换，实现通用AI与医学知识的模型拼接，无需额外训练即可提升医学任务性能，并为单模态视觉编码器引入零样本分类方法，性能接近医学专用多模态方案。
摘要：
通用AI模型，特别是为文本和视觉设计的模型，在广泛的深度学习任务中展现出令人印象深刻的多功能性。然而，它们在医学成像等专业领域往往表现不佳，通常需要特定领域的解决方案或替代的知识转移方法。最近的研究注意到，通用模型在处理语义相关的数据时可能表现出相似的潜在空间，尽管这种对齐不会自然发生。基于这一见解，已有研究表明，应用一种简单的变换——至多为仿射变换——从语义对应样本子集（称为锚点）中估计出来，能够实现跨不同训练范式、架构和模态的模型拼接。在本文中，我们探索语义对齐——估计锚点之间的变换——如何将通用AI与专业医学知识桥接起来。使用多个公开的胸部X射线数据集，我们证明跨模型架构的模型拼接允许通用模型无需额外训练即可整合特定领域的知识，从而在医学任务上提升性能。此外，我们为单模态视觉编码器引入了一种新颖的零样本分类方法，该方法利用了跨模态的语义对齐。我们的结果表明，我们的方法不仅优于通用多模态模型，而且接近完全训练的、医学专用的多模态解决方案的性能水平。
英文摘要：
General-purpose AI models, particularly those designed for text and vision, demonstrate impressive versatility across a wide range of deep-learning tasks. However, they often underperform in specialised domains like medical imaging, where domain-specific solutions or alternative knowledge transfer approaches are typically required. Recent studies have noted that general-purpose models can exhibit similar latent spaces when processing semantically related data, although this alignment does not occur naturally. Building on this insight, it has been shown that applying a simple transformation - at most affine - estimated from a subset of semantically corresponding samples, known as anchors, enables model stitching across diverse training paradigms, architectures, and modalities. In this paper, we explore how semantic alignment - estimating transformations between anchors - can bridge general-purpose AI with specialised medical knowledge. Using multiple public chest X-ray datasets, we demonstrate that model stitching across model architectures allows general models to integrate domain-specific knowledge without additional training, leading to improved performance on medical tasks. Furthermore, we introduce a novel zero-shot classification approach for unimodal vision encoders that leverages semantic alignment across modalities. Our results show that our method not only outperforms general multimodal models but also approaches the performance levels of fully trained, medical-specific multimodal solutions
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
