EventHallusion：诊断视频大语言模型中的事件幻觉问题
EventHallusion: Diagnosing Event Hallucinations in Video LLMs
AI总结：
本文提出事件幻觉评估基准EventHallusion与时序对比解码方法TCD，评测发现开源视频大语言模型事件幻觉问题严重，TCD可有效提升其相关性能。
摘要：
近期，多模态大语言模型（MLLMs）在视频理解领域取得了显著进展。尽管这类视频大语言模型（VideoLLMs）展现出出色的内容推理与指令遵循能力，但其幻觉问题相较于图像域的同类问题尚未得到充分研究。为填补这一空白，我们提出了全新基准EventHallusion，重点评估VideoLLMs在视频分析核心——事件层面的幻觉情况。从幻觉归因角度出发，我们构建的EventHallusion基准可评估VideoLLM受语言先验与视觉-语言偏差影响的程度。此外，我们还提出了一种简单有效的方法——时序对比解码（TCD），用于解决VideoLLMs的幻觉问题。所提TCD方法通过对比原始视频与时序线索被打乱的修改版本视频，在解码阶段修正模型对其先验的偏差。我们在提出的EventHallusion基准上对8个开源和2个闭源VideoLLMs开展了全面评估，发现开源模型的幻觉问题十分严重，而闭源模型的表现明显更优。进一步为开源VideoLLMs配备所提TCD方法后，其在EventHallusion基准的绝大多数指标上均取得了显著性能提升。我们的代码与基准数据可在https://github.com/Stevetich/EventHallusion获取。
英文摘要：
Recently, Multimodal Large Language Models (MLLMs) have made significant progress in the video comprehension field. Despite remarkable content reasoning and instruction following capabilities they demonstrated, the hallucination problem of these VideoLLMs is less explored compared with its counterpart in the image domain. To mitigate this gap, we propose EventHallusion, a novel benchmark that focuses on assessing the VideoLLMs' hallucination toward event, the crux of video analysis. From a hallucination attribution perspective, our EventHallusion benchmark is curated to assess a VideoLLM's susceptibility toward language priors and vision-language biases. On the other hand, we also propose a simple yet effective method, called Temporal Contrastive Decoding (TCD), to tackle the hallucination problems of VideoLLMs. The proposed TCD method rectifies the model's bias toward its priors during the decoding stage by comparing the original video with a modified version, in which temporal cues are disrupted. Through comprehensive evaluation of eight open-source and two closed-source VideoLLMs on the proposed EventHallusion benchmark, we observe that the open-source models suffer significantly from hallucination problems, whereas the closed-source ones perform markedly better. By further equipping open-source VideoLLMs with the proposed TCD approach, evident performance improvements are achieved across most metrics in the EventHallusion benchmark. Our codes and benchmark data are available at https://github.com/Stevetich/EventHallusion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
