arXiv 2603.194822026-04-24cs.CV

无需指令的大型视觉语言模型在医疗指令遵循中的调优

Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following

Department of Electrical and Computer Engineering, The University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院机器人与机电工程系）；Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST)（大邱庆北科学技术院智能机器人系）；Vector Institute（向量研究所）；Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH)（釜山科学技术大学计算机科学与工程系）

Myeongkyun Kang, Soopil Kim, Xiaoxiao Li, Sang Hyun Park

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出无需指令的调优方法，利用图像描述对训练模型，提升医疗领域视觉语言模型的指令遵循能力，实现多选视觉问答任务的最优性能。

摘要：

大型视觉语言模型（LVLMs）在广泛任务中表现出色，其能力主要源于视觉指令调优，即在由精心编纂的图像-指令-输出三元组数据集上微调模型。然而，在医疗领域，构建大规模高质量指令数据集尤为困难，因为需要专门的专家知识。为了解决这个问题，我们提出了一种无需指令的调优方法，减少了对人工指令的依赖，仅利用图像描述对进行微调。具体来说，我们引入了动量代理指令作为替代，保留了预训练LVLM的指令遵循能力，同时促进在推理期间仍有效的参数更新。因此，微调后的LVLM能够灵活响应领域特定指令，即使在微调过程中没有显式指令。此外，我们还结合了响应洗牌策略，以减轻模型对先前词语的过度依赖，从而更有效地进行微调。我们的方法在SKINCON、WBCAtt、CBIS和MIMIC-CXR数据集上的多选视觉问答任务中实现了最先进的准确性，显著提高了LVLM在医疗领域的微调效率。

英文摘要：

Large vision language models (LVLMs) have demonstrated impressive performance across a wide range of tasks. These capabilities largely stem from visual instruction tuning, which fine-tunes models on datasets consisting of curated image-instruction-output triplets. However, in the medical domain, constructing large-scale, high-quality instruction datasets is particularly challenging due to the need for specialized expert knowledge. To address this issue, we propose an instruction-free tuning approach that reduces reliance on handcrafted instructions, leveraging only image-description pairs for fine-tuning. Specifically, we introduce a momentum proxy instruction as a replacement for curated text instructions, which preserves the instruction-following capability of the pre-trained LVLM while promoting updates to parameters that remain valid during inference. Consequently, the fine-tuned LVLM can flexibly respond to domain-specific instructions, even though explicit instructions are absent during fine-tuning. Additionally, we incorporate a response shuffling strategy to mitigate the model's over-reliance on previous words, facilitating more effective fine-tuning. Our approach achieves state-of-the-art accuracy on multiple-choice visual question answering tasks across SKINCON, WBCAtt, CBIS, and MIMIC-CXR datasets, significantly enhancing the fine-tuning efficiency of LVLMs in medical domains.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。