MMedPO：通过临床感知的多模态偏好优化对齐医学视觉-语言模型
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
AI总结：
MMedPO通过引入合理幻觉和病灶区域忽略两类非偏好样本，并结合多模型临床相关性评分加权偏好优化，显著提升了医学视觉-语言模型的事实准确性。
摘要：
大型视觉-语言模型（LVLMs）的进步推动了其在医学领域的应用。然而，医学LVLMs（Med-LVLMs）由于模态错位而面临事实性挑战，即模型优先考虑文本知识而非视觉输入，导致产生与医学图像信息相矛盾的幻觉。以往通过偏好优化来增强Med-LVLMs模态对齐的尝试，未能充分缓解偏好数据中的临床相关性不足问题，使得这些样本容易被区分，从而降低了对齐效果。为解决这一挑战，我们提出了MMedPO，一种新颖的多模态医学偏好优化方法，该方法考虑偏好样本的临床相关性以增强Med-LVLM对齐。MMedPO通过引入两种类型的非偏好样本来构建多模态偏好数据：（1）通过目标Med-LVLMs或GPT-4o注入的合理幻觉，生成医学上不准确的响应；（2）通过局部病灶加噪实现的病灶区域忽略，破坏对关键区域的视觉理解。然后，我们基于多个Med-LLMs和视觉工具的评分计算每个样本的临床相关性，并将这些评分作为权重整合到偏好优化过程中，从而实现有效对齐。我们的实验表明，MMedPO显著增强了Med-LVLMs的事实准确性，在Med-VQA和报告生成任务上分别平均比现有偏好优化方法提升了14.2%和51.7%。我们的代码可在https://github.com/aiming-lab/MMedPO获取。
英文摘要：
The advancement of Large Vision-Language Models (LVLMs) has propelled their application in the medical field. However, Medical LVLMs (Med-LVLMs) encounter factuality challenges due to modality misalignment, where the models prioritize textual knowledge over visual input, leading to hallucinations that contradict information in medical images. Previous attempts to enhance modality alignment in Med-LVLMs through preference optimization have inadequately mitigated clinical relevance in preference data, making these samples easily distinguishable and reducing alignment effectiveness. To address this challenge, we propose MMedPO, a novel multimodal medical preference optimization approach that considers the clinical relevance of preference samples to enhance Med-LVLM alignment. MMedPO curates multimodal preference data by introducing two types of dispreference: (1) plausible hallucinations injected through target Med-LVLMs or GPT-4o to produce medically inaccurate responses, and (2) lesion region neglect achieved through local lesion-noising, disrupting visual understanding of critical areas. We then calculate clinical relevance for each sample based on scores from multiple Med-LLMs and visual tools, and integrate these scores into the preference optimization process as weights, enabling effective alignment. Our experiments demonstrate that MMedPO significantly enhances factual accuracy in Med-LVLMs, achieving substantial improvements over existing preference optimization methods by averaging 14.2% and 51.7% across the Med-VQA and report generation tasks. Our code are available in https://github.com/aiming-lab/MMedPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
