IRIS: 隐式奖励引导的内部筛选以缓解多模态幻觉
IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination
AI总结：
IRIS通过隐式奖励引导内部筛选，有效缓解多模态大语言模型的幻觉问题，无需外部反馈且性能优异。
摘要：
幻觉仍然是多模态大语言模型（MLLMs）中的根本性挑战。尽管直接偏好优化（DPO）是关键对齐框架，但现有方法往往依赖于昂贵的外部评估器进行评分或重写，导致离策略学习间隙和离散化损失。由于缺乏对内部状态的访问，此类反馈忽略了在生成过程中不同模态之间导致幻觉的细粒度冲突。为了解决这个问题，我们提出了IRIS（隐式奖励引导的内部筛选），它利用原生对数概率空间中的连续隐式奖励来保持完整的信息密度并捕捉内部模态竞争。这种在线策略范式通过利用自动生成的偏好对消除了学习间隙。通过基于多模态隐式奖励筛选这些对，IRIS确保优化由直接解决模态冲突的信号驱动。广泛的实验表明，IRIS仅使用5.7k个样本，在关键幻觉基准上实现了高度竞争性的性能，无需在偏好对齐过程中使用任何外部反馈。这些结果证实，IRIS提供了一种高效且原则性的范式，用于缓解MLLMs的幻觉。
英文摘要：
Hallucination remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While Direct Preference Optimization (DPO) is a key alignment framework, existing approaches often rely heavily on costly external evaluators for scoring or rewriting, incurring off-policy learnability gaps and discretization loss. Due to the lack of access to internal states, such feedback overlooks the fine-grained conflicts between different modalities that lead to hallucinations during generation. To address this issue, we propose IRIS (Implicit Reward-Guided Internal Sifting), which leverages continuous implicit rewards in the native log-probability space to preserve full information density and capture internal modal competition. This on-policy paradigm eliminates learnability gaps by utilizing self-generated preference pairs. By sifting these pairs based on multimodal implicit rewards, IRIS ensures that optimization is driven by signals that directly resolve modal conflicts. Extensive experiments demonstrate that IRIS achieves highly competitive performance on key hallucination benchmarks using only 5.7k samples, without requiring any external feedback during preference alignment. These results confirm that IRIS provides an efficient and principled paradigm for mitigating MLLM hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
