arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00400cs.AI
收藏

KEPO:基于知识的偏好优化用于多模态推理及其在医学VQA中的应用

KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQA

Chapman University(查普曼大学);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of California, Irvine(加州大学伊文斯分校)

Fan Yang, Rui Meng, Trudi Di Qi, Ali Ezzati, Yuxin Wen

AI总结:

KEPO通过质量门控的在线蒸馏和知识增强的探索策略,提升多模态推理任务的训练稳定性与推理一致性,优于强化学习和在线蒸馏基线。

摘要:

强化学习(RL)已成为诱导大语言和视觉-语言模型显式推理行为的有希望范式。然而,训练后的推理导向RL仍面临根本挑战,由于稀疏轨迹级奖励导致信用分配模糊和严重探索失败,可能使策略陷入『学习悬崖』。最近的在线蒸馏方法引入密集教师监督以稳定优化,但将其统一应用于所有生成轨迹。我们认为这种统一蒸馏不适合推理密集型任务,因为低质量在线轨迹常源于早期逻辑错误,而蒸馏在错误上下文中会注入噪声和对齐不良的梯度。为解决这些挑战,我们提出了知识增强的偏好优化(KEPO),一个统一的训练后框架,整合:(i)一个质量门控的在线蒸馏目标,仅对高质量轨迹应用密集教师指导;(ii)一个知识增强的探索策略,利用教师模型学习的提示来拒绝性采样奖励积极的在线轨迹以RL,从而缓解探索崩溃。在具有挑战性的医学视觉问答基准测试中,单源泛化条件下,KEPO展示了改进的训练稳定性、更连贯的推理行为和优于强化学习和在线蒸馏基线的分布外性能。

英文摘要:

Reinforcement learning (RL) has emerged as a promising paradigm for inducing explicit reasoning behaviors in large language and vision-language models. However, reasoning-oriented RL post-training remains fundamentally challenging due to sparse trajectory-level rewards, leading to ambiguous credit assignment and severe exploration failures that can trap the policy in a ``learning cliff.'' Recent on-policy distillation methods introduce dense teacher supervision to stabilize optimization, but apply it uniformly across all generated trajectories. We argue that such uniform distillation is ill-suited for reasoning-intensive tasks, as low-quality on-policy trajectories often originate from early logical errors, and distillation under flawed contexts injects noisy and misaligned gradients. To address these challenges, we propose Knowledge-Enhanced Preference Optimization (KEPO), a unified post-training framework that integrates: (i) a quality-gated on-policy distillation objective that selectively applies dense teacher guidance only to high-quality trajectories, and (ii) a knowledge-enhanced exploration strategy that leverages hints learned from a teacher model to rejectively sample reward-positive on-policy trajectories for RL, thereby mitigating exploration collapse. Evaluated on a challenging medical visual question answering benchmark under single-source generalization, KEPO demonstrates improved training stability, more coherent reasoning behaviors, and superior out-of-distribution performance over reinforcement learning and on-policy distillation baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑