EAGLE：增强型视觉定位可最小化指令多模态模型中的幻觉
EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models
AI总结：
本文提出 EAGLE，通过重构对比预训练任务增强视觉编码器的定位与语言对齐，在不依赖 LLM 或融合模块、无需额外指令训练的情况下显著降低多模态模型幻觉。
摘要：
大型语言模型和视觉 Transformer 已展现出令人瞩目的零样本能力，从而在下游任务中实现了显著的可迁移性。这些模型的融合产生了具有更强指令能力的多模态架构。尽管这些多模态架构纳入了大规模图像和语言预训练，但它们仍经常生成偏离图像数据中真实答案的响应。这些失败案例被称为幻觉。当前用于缓解幻觉的方法通常聚焦于正则化语言组件、改进融合模块，或集成多个视觉编码器以提升视觉表示。本文通过直接增强视觉组件的能力来解决幻觉问题。我们提出的方法名为 EAGLE，它完全不依赖于 LLM 或融合模块，并作为一种预训练后方法发挥作用，用于改进视觉编码器的定位和语言对齐。我们表明，对原始对比预训练任务进行直接重构，即可得到一个改进的视觉编码器；该编码器能够被纳入指令多模态架构，而无需额外的指令训练。因此，EAGLE 在多个具有挑战性的基准和任务中显著减少了幻觉。
英文摘要：
Large language models and vision transformers have demonstrated impressive zero-shot capabilities, enabling significant transferability in downstream tasks. The fusion of these models has resulted in multi-modal architectures with enhanced instructional capabilities. Despite incorporating vast image and language pre-training, these multi-modal architectures often generate responses that deviate from the ground truth in the image data. These failure cases are known as hallucinations. Current methods for mitigating hallucinations generally focus on regularizing the language component, improving the fusion module, or ensembling multiple visual encoders to improve visual representation. In this paper, we address the hallucination issue by directly enhancing the capabilities of the visual component. Our approach, named EAGLE, is fully agnostic to the LLM or fusion module and works as a post-pretraining approach that improves the grounding and language alignment of the visual encoder. We show that a straightforward reformulation of the original contrastive pre-training task results in an improved visual encoder that can be incorporated into the instructional multi-modal architecture without additional instructional training. As a result, EAGLE achieves a significant reduction in hallucinations across multiple challenging benchmarks and tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
