VL-GenRM：通过视觉专家与迭代训练增强视觉-语言验证
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
AI总结：
针对视觉-语言奖励模型训练中的自举困境、模态偏差等问题，提出融合视觉专家、思维链与边际拒绝采样的迭代训练框架，提升幻觉检测与多模态推理性能，推进VL模型强化学习对齐。
摘要：
带有可验证奖励的强化微调（RFT）已推动大语言模型取得进展，但在视觉-语言（VL）模型领域的研究仍不充分。视觉-语言奖励模型（VL-RM）通过提供结构化反馈实现VL模型对齐，是该过程的核心，但训练有效的VL-RM面临两大挑战。其一为自举困境：高质量训练数据依赖已具备较强能力的VL模型，形成自生成监督强化现有偏差的循环。其二是模态偏差与负例放大：VL模型产生错误视觉属性的幻觉时，会生成有缺陷的偏好数据，进一步误导训练。为解决这些问题，我们提出一种迭代训练框架，该框架利用视觉专家、思维链（Chain-of-Thought, CoT）推理依据以及基于边际的拒绝采样方法。我们的方法优化了偏好数据集，增强了结构化批评能力，并能迭代提升推理水平。在多个VL-RM基准上的实验表明，该方法在幻觉检测和多模态推理方面表现优异，推动了基于强化学习的VL模型对齐研究。
英文摘要：
Reinforcement Fine-Tuning (RFT) with verifiable rewards has advanced large language models but remains underexplored for Vision-Language (VL) models. The Vision-Language Reward Model (VL-RM) is key to aligning VL models by providing structured feedback, yet training effective VL-RMs faces two major challenges. First, the bootstrapping dilemma arises as high-quality training data depends on already strong VL models, creating a cycle where self-generated supervision reinforces existing biases. Second, modality bias and negative example amplification occur when VL models hallucinate incorrect visual attributes, leading to flawed preference data that further misguides training. To address these issues, we propose an iterative training framework leveraging vision experts, Chain-of-Thought (CoT) rationales, and Margin-based Rejection Sampling. Our approach refines preference datasets, enhances structured critiques, and iteratively improves reasoning. Experiments across VL-RM benchmarks demonstrate superior performance in hallucination detection and multimodal reasoning, advancing VL model alignment with reinforcement learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
