arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.02458eess.IVcs.CLcs.CV
收藏

MedVisionLlama:利用预训练大语言模型层增强医学图像分割

MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation

Montreal Neurological Institute, McGill University(蒙特利尔神经科学研究所,麦吉尔大学)

Gurucharan Marthi Krishna Kumar, Aman Chadha, Janine Mendola, Amir Shmuel

AI总结:

MedVisionLlama通过将冻结的预训练LLM transformer模块集成到ViT编码器中,并结合混合注意力与多尺度融合机制,在多种医学成像模态上显著提升了分割性能,平均Dice分数从0.74提升至0.79。

摘要:

大语言模型(LLMs)以其在文本数据上的多功能性而闻名,人们正日益探索其在增强医学图像分割方面的潜力,而医学图像分割是实现精确诊断成像的一项关键任务。本研究探索通过集成预训练的LLM transformer模块来增强用于医学图像分割的视觉Transformer(ViTs)。我们的方法将一个冻结的LLM transformer模块整合到基于ViT的模型的编码器中,从而在多种医学成像模态下显著提升了分割性能。我们提出了一种混合注意力机制(Hybrid Attention Mechanism),将全局与局部特征学习相结合,并采用多尺度融合模块(Multi-Scale Fusion Block)来聚合不同尺度上的特征。增强后的模型表现出显著的性能提升,包括平均Dice分数从0.74提高到0.79,以及准确率、精确率和Jaccard指数的改善。这些结果证明了基于LLM的transformer在优化医学图像分割方面的有效性,凸显了其在显著提升模型准确性和鲁棒性方面的潜力。源代码及我们的实现可在以下地址获取:https://github.com/AS-Lab/Marthi-et-al-2025-MedVisionLlama-Pre-Trained-LLM-Layers-to-Enhance-Medical-Image-Segmentation

英文摘要:

Large Language Models (LLMs), known for their versatility in textual data, are increasingly being explored for their potential to enhance medical image segmentation, a crucial task for accurate diagnostic imaging. This study explores enhancing Vision Transformers (ViTs) for medical image segmentation by integrating pre-trained LLM transformer blocks. Our approach, which incorporates a frozen LLM transformer block into the encoder of a ViT-based model, leads to substantial improvements in segmentation performance across various medical imaging modalities. We propose a Hybrid Attention Mechanism that combines global and local feature learning with a Multi-Scale Fusion Block for aggregating features across different scales. The enhanced model shows significant performance gains, including an average Dice score increase from 0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index. These results demonstrate the effectiveness of LLM-based transformers in refining medical image segmentation, highlighting their potential to significantly boost model accuracy and robustness. The source code and our implementation are available at: https://github.com/AS-Lab/Marthi-et-al-2025-MedVisionLlama-Pre-Trained-LLM-Layers-to-Enhance-Medical-Image-Segmentation

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑