MoD-DPO：通过模态解耦偏好优化缓解多模态幻觉
MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
AI总结：
本文提出MoD-DPO框架，通过引入模态感知正则化项和语言先验去偏惩罚，提升多模态大模型的模态对齐能力，实验表明其在多模态幻觉基准测试中表现优异。
摘要：
多模态大语言模型（多模态LLM）在音频视觉理解任务中表现强劲，但易受跨模态幻觉影响。本文提出Modality-Decoupled Direct Preference Optimization（MoD-DPO），通过引入模态感知正则化项和语言先验去偏惩罚，减少无关模态的干扰和相关模态的敏感性，从而降低意外的跨模态交互。实验表明，MoD-DPO在多个多模态幻觉基准测试中提升了感知准确性和幻觉抵抗能力，优于之前的偏好优化基线。
英文摘要：
Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
