arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.07613cs.CV
收藏

数据高效的视觉语言模型微调用于阿尔茨海默病诊断

Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease

School of Mathematics and Statistics, University of Glasgow, UK(数学与统计学学院,格拉斯哥大学)

Fangqi Cheng, Surajit Ray, Xiaochen Yang

AI总结:

提出数据高效的微调流程,将3D CT医学视觉语言模型适配至3D MRI,通过合成报告和MMSE辅助令牌实现轻量提示调优,在AD诊断中以少量数据达到最先进性能。

摘要:

医学视觉语言模型(Med-VLMs)在报告生成和视觉问答等任务中表现出色,但仍面临若干限制。最显著的是,它们未充分利用患者元数据,且缺乏临床诊断知识的整合。此外,现有大多数模型通常从头训练或在大规模2D图像-文本对上进行微调,需要大量计算资源,并且由于缺乏结构信息,它们在3D医学影像上的有效性往往受限。为解决这些不足,我们提出了一种数据高效的微调流程,将基于3D CT的Med-VLMs适配到3D MRI,并展示了其在阿尔茨海默病(AD)诊断中的应用。我们的系统引入了两项关键创新。首先,我们将结构化元数据转换为合成报告,丰富文本输入以改善图像-文本对齐。其次,我们添加了一个辅助令牌,训练其预测简易精神状态检查(MMSE)评分,这是一种广泛使用的认知功能临床指标,与AD严重程度相关。这为微调提供了额外的监督。通过对图像和文本模态应用轻量级提示调优,我们的方法在ADNI上仅用1,504个训练MRI即达到最先进性能,优于在27,161个MRI上训练的方法,并在OASIS-2和AIBL上展现出强大的零样本泛化能力。代码可在https://github.com/CFQ666312/DEFT-VLM-AD获取。

英文摘要:

Medical vision-language models (Med-VLMs) have shown impressive results in tasks such as report generation and visual question answering, but they still face several limitations. Most notably, they underutilize patient metadata and lack integration of clinical diagnostic knowledge. Moreover, most existing models are typically trained from scratch or fine-tuned on large-scale 2D image-text pairs, requiring extensive computational resources, and their effectiveness on 3D medical imaging is often limited due to the absence of structural information. To address these gaps, we propose a data-efficient fine-tuning pipeline to adapt 3D CT-based Med-VLMs for 3D MRI and demonstrate its application in Alzheimer's disease (AD) diagnosis. Our system introduces two key innovations. First, we convert structured metadata into synthetic reports, enriching textual input for improved image-text alignment. Second, we add an auxiliary token trained to predict the mini-mental state examination (MMSE) score, a widely used clinical measure of cognitive function that correlates with AD severity. This provides additional supervision for fine-tuning. Applying lightweight prompt tuning to both image and text modalities, our approach achieves state-of-the-art performance on ADNI with only 1,504 training MRIs, outperforming methods trained on 27,161 MRIs, and shows strong zero-shot generalization on OASIS-2 and AIBL. Code is available at https://github.com/CFQ666312/DEFT-VLM-AD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑