LVLMs中的语言偏差：从深入分析到简单有效的缓解方法
Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
AI总结：
本文系统研究了大视觉语言模型中的语言偏差问题，发现其根源在于训练中的模态未对齐，并提出了两种简单有效的缓解方法：语言偏差正则化（LBR）和语言偏差惩罚（LBP）。
摘要：
大型视觉语言模型（LVLMs）通过视觉理解扩展了大型语言模型，但仍然容易产生幻觉，即输出流畅但与图像不一致。最近的研究将这一问题与语言偏差联系起来——LVLMs过度依赖文本而忽视视觉输入的倾向。然而，大多数分析仍然是经验性的，没有揭示其根本原因。在本文中，我们对语言偏差进行了系统研究，并确定其根源在于训练过程中的模态未对齐。我们的分析表明，视觉指令微调（VIT）和直接偏好优化（DPO）通常优先考虑文本改进，这可能导致LVLMs过度倾向于语言建模，而不是平衡的多模态理解。为了解决这个问题，我们提出了两种简单而有效的方法：语言偏差正则化（LBR），通过在指令微调期间进行正则化来缓解语言偏差；以及语言偏差惩罚（LBP），在DPO训练过程中惩罚语言偏差。跨多种模型和基准的大量实验证明了我们方法的有效性。LBR在十多个通用基准上持续提高性能，而LBP显著减少了幻觉并提高了可信度。这些方法共同不仅缓解了语言偏差，还促进了LVLMs的整体对齐，且无需引入任何额外数据或辅助模型。我们的代码公开在https://github.com/lab-klc/LVLM-Language-Bias。
英文摘要：
Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to language bias-the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: Language Bias Regularization (LBR) which mitigates language bias through regularization during instruction tuning, and Language Bias Penalty (LBP), which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/LVLM-Language-Bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
