在所见之地接地：通过标题反馈、多样性感知采样和冲突正则化实现抗幻觉的MLLMs
Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization
AI总结：
本文提出抗幻觉的MLLMs方法，通过标题反馈、多样性感知采样和冲突正则化减少幻觉，提升推理准确性。
摘要：
尽管多模态大语言模型（MLLMs）在多样任务上取得了显著成功，但其实际部署受到幻觉问题的严重阻碍，特别是在强化学习（RL）优化过程中尤为突出。本文系统分析了在RL训练下MLLMs中幻觉的根本原因，识别出三个关键因素：（1）对链式视觉推理的过度依赖，其中不准确的初始描述或冗余信息将后续推断锚定在错误的前提上；（2）在策略优化过程中探索多样性不足，导致模型生成过于自信但错误的输出；（3）训练样本之间的破坏性冲突，其中神经切线核（NTK）相似性导致虚假关联和参数更新不稳定。为解决这些挑战，我们提出一个包含三个核心模块的综合框架。首先，我们通过在推理阶段之前引入专门的规划和标题生成阶段来增强视觉定位，采用基于质量的标题奖励来确保准确的初始锚定。其次，为了提高探索性，我们根据样本奖励分布的均值和方差对样本进行分类，优先选择高方差的样本以使模型专注于多样且信息丰富的数据。最后，为了缓解样本干扰，我们通过将样本对分组并应用InfoNCE损失来调节NTK相似性，从而将过于相似的样本推远，将不相似的样本拉近，从而引导梯度交互朝着平衡的范围发展。实验结果表明，我们提出的方法显著降低了幻觉率，并有效提高了MLLMs的推理准确性。
英文摘要：
While Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse tasks, their practical deployment is severely hindered by hallucination issues, which become particularly acute during Reinforcement Learning (RL) optimization. This paper systematically analyzes the root causes of hallucinations in MLLMs under RL training, identifying three critical factors: (1) an over-reliance on chained visual reasoning, where inaccurate initial descriptions or redundant information anchor subsequent inferences to incorrect premises; (2) insufficient exploration diversity during policy optimization, leading the model to generate overly confident but erroneous outputs; and (3) destructive conflicts between training samples, where Neural Tangent Kernel (NTK) similarity causes false associations and unstable parameter updates. To address these challenges, we propose a comprehensive framework comprising three core modules. First, we enhance visual localization by introducing dedicated planning and captioning stages before the reasoning phase, employing a quality-based caption reward to ensure accurate initial anchoring. Second, to improve exploration, we categorize samples based on the mean and variance of their reward distributions, prioritizing samples with high variance to focus the model on diverse and informative data. Finally, to mitigate sample interference, we regulate NTK similarity by grouping sample pairs and applying an InfoNCE loss to push overly similar pairs apart and pull dissimilar ones closer, thereby guiding gradient interactions toward a balanced range. Experimental results demonstrate that our proposed method significantly reduces hallucination rates and effectively enhances the inference accuracy of MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
