缓解多模态大语言模型在序列图像上的行为幻觉
Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
AI总结：
针对序列图像中研究不足的行为幻觉，本文提出轻量级两阶段框架 SHE，通过自适应时间窗口对齐检测与联合嵌入空间正交投影缓解幻觉，并用新指标 BEACH 验证其可降低幻觉逾10%。
摘要：
尽管多模态大语言模型擅长各类任务，但它们仍受到幻觉困扰，这限制了其在更广泛领域应用中的可靠性和可扩展性。为解决该问题，近期研究主要关注客观幻觉。然而，对于序列图像，除客观幻觉外，还存在研究较少的行为幻觉。本工作旨在填补这一空白。我们首先揭示行为幻觉主要源于两个关键因素：先验驱动偏差和雪球效应。基于这些观察，我们提出 SHE（Sequence Hallucination Eradication，序列幻觉消除），这是一个轻量级两阶段框架：（1）使用我们提出的自适应时间窗口，通过视觉-文本对齐检查检测幻觉；（2）通过在联合嵌入空间上进行正交投影来缓解幻觉。我们还提出了一种新指标 BEACH，用于量化行为幻觉的严重程度。在标准基准上的实验结果表明，SHE 在保持描述准确性的同时，在 BEACH 上将行为幻觉降低了 10% 以上。
英文摘要：
While multimodal large language models excel at various tasks, they still suffer from hallucinations, which limit their reliability and scalability for broader domain applications. To address this issue, recent research mainly focuses on objective hallucination. However, for sequential images, besides objective hallucination, there is also behavioral hallucination, which is less studied. This work aims to fill in the gap. We first reveal that behavioral hallucinations mainly arise from two key factors: prior-driven bias and the snowball effect. Based on these observations, we introduce SHE (Sequence Hallucination Eradication), a lightweight, two-stage framework that (1) detects hallucinations via visual-textual alignment check using our proposed adaptive temporal window and (2) mitigates them via orthogonal projection onto the joint embedding space. We also propose a new metric (BEACH) to quantify behavioral hallucination severity. Empirical results on standard benchmarks demonstrate that SHE reduces behavioral hallucination by over 10% on BEACH while maintaining descriptive accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
