RARL：在数据与硬件约束下通过强化学习和LoRA提升医学视觉语言模型的推理与泛化能力
RARL: Improving Medical VLM Reasoning and Generalization with Reinforcement Learning and LoRA under Data and Hardware Constraints
AI总结：
针对医学VLM泛化性、效率不足难落地的问题，提出RARL推理感知强化学习框架，结合LoRA和自定义奖励微调轻量模型，单GPU即可训练，在推理任务和跨数据集泛化上均优于监督及传统RL微调方法。
摘要：
视觉语言模型（VLM）在医学应用中的日益普及为诊断推理提供了颇具前景的支持。然而，当前的医学VLM通常在泛化性、透明度和计算效率方面存在局限，这些障碍阻碍了其在资源受限的真实场景中的部署。为应对这些挑战，我们提出了一种推理感知强化学习框架——RARL，该框架可增强医学VLM的推理能力，同时保持高效性并能适配低资源环境。我们的方法使用低秩适配（Low-Rank Adaptation，即LoRA）和同时兼顾诊断准确性与推理质量的自定义奖励函数，对轻量级基础模型Qwen2-VL-2B-Instruct进行微调。训练仅在单张NVIDIA A100-PCIE-40GB GPU上完成，证明了此类模型在受限环境中部署的可行性。我们采用LLM-as-judge框架对模型进行评估，该框架会同时为答案正确性和解释质量打分。实验结果表明，RARL显著提升了VLM在医学图像分析和临床推理中的性能，在聚焦推理的任务上相比监督微调性能提升约7.78%，同时所需计算资源更少。此外，我们验证了该方法在未见过的数据集上的泛化能力，相比监督微调性能提升约27%，相比传统强化学习微调提升约4%。我们的实验还表明，训练阶段的多样性提示和推理阶段的推理提示对提升VLM性能至关重要。研究结果凸显了推理引导学习和推理提示在推动医学VLM迈向更透明、准确且资源高效的临床决策方面的潜力。代码和数据已公开。
英文摘要：
The growing integration of vision-language models (VLMs) in medical applications offers promising support for diagnostic reasoning. However, current medical VLMs often face limitations in generalization, transparency, and computational efficiency-barriers that hinder deployment in real-world, resource-constrained settings. To address these challenges, we propose a Reasoning-Aware Reinforcement Learning framework, \textbf{RARL}, that enhances the reasoning capabilities of medical VLMs while remaining efficient and adaptable to low-resource environments. Our approach fine-tunes a lightweight base model, Qwen2-VL-2B-Instruct, using Low-Rank Adaptation and custom reward functions that jointly consider diagnostic accuracy and reasoning quality. Training is performed on a single NVIDIA A100-PCIE-40GB GPU, demonstrating the feasibility of deploying such models in constrained environments. We evaluate the model using an LLM-as-judge framework that scores both correctness and explanation quality. Experimental results show that RARL significantly improves VLM performance in medical image analysis and clinical reasoning, outperforming supervised fine-tuning on reasoning-focused tasks by approximately 7.78%, while requiring fewer computational resources. Additionally, we demonstrate the generalization capabilities of our approach on unseen datasets, achieving around 27% improved performance compared to supervised fine-tuning and about 4% over traditional RL fine-tuning. Our experiments also illustrate that diversity prompting during training and reasoning prompting during inference are crucial for enhancing VLM performance. Our findings highlight the potential of reasoning-guided learning and reasoning prompting to steer medical VLMs toward more transparent, accurate, and resource-efficient clinical decision-making. Code and data are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
