聚焦效率：LayerNorm作为微调医学视觉语言预训练模型的催化剂
Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models
AI总结：
本文探索微调Med-VLMs内在组件的影响，发现仅微调LayerNorm层在医学视觉问答和影像报告生成等任务中，效率超越传统PEFT方法且保持准确性与泛化能力，尤其适用于大规模模型。
摘要：
在医学视觉语言模型领域，对通用高效微调机制的追求至关重要，特别是考虑到跨学科领域的研究人员通常极度缺乏训练资源，然而这一问题在很大程度上尚未被探索。鉴于医学领域的独特挑战，如数据范围有限和显著的领域特定要求，专门针对Med-VLMs评估和适配参数高效微调（PEFT）方法是必不可少的。目前Med-VLMs上的大多数PEFT方法尚未得到全面研究，而是主要集中在向模型结构或输入中添加一些组件。然而，微调模型内在组件通常能产生更好的通用性和一致性，其对Med-VLMs最终性能的影响被广泛忽视且研究不足。本文致力于探索传统PEFT方法的替代方案，特别是微调LayerNorm层、FFNs和Attention层对Med-VLMs的影响。我们的综合研究涵盖了小规模和大规模的Med-VLMs，评估了它们在医学视觉问答和医学影像报告生成等任务中各种微调范式下的性能。研究结果揭示了对Med-VLMs进行下游任务微调时内在参数微调方法的独特见解，并发现仅微调LayerNorm层不仅超越了传统PEFT方法的效率，还在一系列医学下游任务中保持了模型的准确性和泛化能力。实验表明LayerNorm微调具有卓越的适应性和可扩展性，特别是在大规模Med-VLMs的背景下。
英文摘要：
In the realm of Medical Visual Language Models (Med-VLMs), the quest for universal efficient fine-tuning mechanisms remains paramount, especially given researchers in interdisciplinary fields are often extremely short of training resources, yet largely unexplored. Given the unique challenges in the medical domain, such as limited data scope and significant domain-specific requirements, evaluating and adapting Parameter-Efficient Fine-Tuning (PEFT) methods specifically for Med-VLMs is essential. Most of the current PEFT methods on Med-VLMs have yet to be comprehensively investigated but mainly focus on adding some components to the model's structure or input. However, fine-tuning intrinsic model components often yields better generality and consistency, and its impact on the ultimate performance of Med-VLMs has been widely overlooked and remains understudied. In this paper, we endeavour to explore an alternative to traditional PEFT methods, especially the impact of fine-tuning LayerNorm layers, FFNs and Attention layers on the Med-VLMs. Our comprehensive studies span both small-scale and large-scale Med-VLMs, evaluating their performance under various fine-tuning paradigms across tasks such as Medical Visual Question Answering and Medical Imaging Report Generation. The findings reveal unique insights into the effects of intrinsic parameter fine-tuning methods on fine-tuning Med-VLMs to downstream tasks and expose fine-tuning solely the LayerNorm layers not only surpasses the efficiency of traditional PEFT methods but also retains the model's accuracy and generalization capabilities across a spectrum of medical downstream tasks. The experiments show LayerNorm fine-tuning's superior adaptability and scalability, particularly in the context of large-scale Med-VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
