ELV-Halluc：长视频理解中的语义聚合幻觉基准测试
ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding
AI总结：
针对长视频理解中帧级语义正确但聚合后出错的语义聚合幻觉（SAH）问题，研究推出首个长视频幻觉基准ELV-Halluc，证实SAH随语义复杂度升高而增加，并通过位置编码优化和DPO策略结合8K对抗数据集将SAH比例降低27.7%。
摘要：
视频多模态大语言模型（Video-MLLMs）在视频理解领域已取得显著进展。然而，这类模型仍易产生幻觉，即生成与视频输入不一致或无关的内容。以往的视频幻觉基准主要聚焦于短视频，将幻觉归因于强语言先验、帧缺失或视觉编码器引入的视觉-语言偏差等因素。尽管这些原因确实能解释短视频中的大部分幻觉，但它们仍过度简化了幻觉的成因。有时，模型生成的输出有误，但帧级语义却是正确的。我们将这类幻觉称为语义聚合幻觉（Semantic Aggregation Hallucination，SAH），它产生于将帧级语义聚合为事件级语义组的过程中。鉴于长视频中多事件的语义复杂度更高，SAH 问题会变得尤为突出，因此有必要将这类幻觉的成因分离出来并进行深入研究。为解决上述问题，我们推出了 ELV-Halluc，这是首个专门针对长视频幻觉的基准，可对 SAH 展开系统性研究。我们的实验证实了 SAH 的存在，并表明其会随语义复杂度的提升而增加。此外，我们还发现模型在语义快速变化的场景下更易产生 SAH。不仅如此，我们还探讨了缓解 SAH 的潜在方法。我们证明位置编码策略有助于减轻 SAH，并进一步采用 DPO 策略来增强模型区分事件内部和跨事件语义的能力。为此，我们整理了一个包含 8K 对抗数据对的数据集，在 ELV-Halluc 和 Video-MME 上均取得了性能提升，其中 SAH 比例大幅下降了 27.7%。
英文摘要：
Video multimodal large language models (Video-MLLMs) have achieved remarkable progress in video understanding. However, they remain vulnerable to hallucination-producing content inconsistent with or unrelated to video inputs. Previous video hallucination benchmarks primarily focus on short-videos. They attribute hallucinations to factors such as strong language priors, missing frames, or vision-language biases introduced by the visual encoder. While these causes indeed account for most hallucinations in short videos, they still oversimplify the cause of hallucinations. Sometimes, models generate incorrect outputs but with correct frame-level semantics. We refer to this type of hallucination as Semantic Aggregation Hallucination (SAH), which arises during the process of aggregating frame-level semantics into event-level semantic groups. Given that SAH becomes particularly critical in long videos due to increased semantic complexity across multiple events, it is essential to separate and thoroughly investigate the causes of this type of hallucination. To address the above issues, we introduce ELV-Halluc, the first benchmark dedicated to long-video hallucination, enabling a systematic investigation of SAH. Our experiments confirm the existence of SAH and show that it increases with semantic complexity. Additionally, we find that models are more prone to SAH on rapidly changing semantics. Moreover, we discuss potential approaches to mitigate SAH. We demonstrate that positional encoding strategy contributes to alleviating SAH, and further adopt DPO strategy to enhance the model's ability to distinguish semantics within and across events. To support this, we curate a dataset of 8K adversarial data pairs and achieve improvements on both ELV-Halluc and Video-MME, including a substantial 27.7% reduction in SAH ratio.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
