arXiv 2506.008052025-06-01cs.CVcs.CL

HSCR：用于对齐医学视觉语言模型的分层自对比奖励方法

HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models

Zhejiang University（浙江大学）；Byte Dance（字节跳动）；National University of Singapore（新加坡国立大学）；Angelalign Inc China（Angelalign中国公司）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）

Songtao Jiang, Yan Zhang, Yeying Jin, Zhihang Tang, Yangyang Wu, Yang Feng, Jian Wu, Zuozhu Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出分层自对比奖励方法HSCR，通过视觉token丢弃识别模态耦合token并生成高质量非偏好数据，结合多级偏好优化策略，在仅2000条训练数据下显著提升医学视觉语言模型的模态对齐与可信度。

摘要：

医学视觉语言模型（Med-VLMs）已在多种任务中取得成功，但现有大多数方法忽视了模态错位问题，该问题可能导致临床环境中产生不可信的回答。本文提出分层自对比奖励（HSCR），一种解决Med-VLM对齐中两个关键挑战的新方法：1）低成本生成高质量偏好数据；2）捕捉细微且上下文感知的偏好以改善对齐。HSCR首先利用Med-VLM的固有能力，以更高的采样概率生成非偏好回答。通过分析视觉token丢弃后的输出logit变化，我们识别出导致错位的模态耦合token，并推导出隐式对齐奖励函数。该函数在解码过程中引导将token替换为幻觉token，从而生成高质量的非偏好数据。此外，HSCR引入了一种多级偏好优化策略，该策略超越了传统的相邻级别优化，通过纳入细微的隐式偏好，利用非偏好数据中的相对质量来捕捉微妙的对齐线索，从而实现更精确且上下文感知的优化。在包括Med-VQA、医学图像描述和指令跟随在内的多个医学任务上的大量实验表明，HSCR不仅提升了零样本性能，而且仅用2,000条训练数据就显著改善了模态对齐和可信度。

英文摘要：

Medical Vision-Language Models (Med-VLMs) have achieved success across various tasks, yet most existing methods overlook the modality misalignment issue that can lead to untrustworthy responses in clinical settings. In this paper, we propose Hierarchical Self-Contrastive Rewarding (HSCR), a novel approach that addresses two critical challenges in Med-VLM alignment: 1) Cost-effective generation of high-quality preference data; 2) Capturing nuanced and context-aware preferences for improved alignment. HSCR first leverages the inherent capability of Med-VLMs to generate dispreferred responses with higher sampling probability. By analyzing output logit shifts after visual token dropout, we identify modality-coupled tokens that induce misalignment and derive an implicit alignment reward function. This function guides token replacement with hallucinated ones during decoding, producing high-quality dispreferred data. Furthermore, HSCR introduces a multi-level preference optimization strategy, which extends beyond traditional adjacent-level optimization by incorporating nuanced implicit preferences, leveraging relative quality in dispreferred data to capture subtle alignment cues for more precise and context-aware optimization. Extensive experiments across multiple medical tasks, including Med-VQA, medical image captioning and instruction following, demonstrate that HSCR not only enhances zero-shot performance but also significantly improves modality alignment and trustworthiness with just 2,000 training entries.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。