EGOILLUSION: 以第一视角视频理解中的幻觉问题基准测试
EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding
AI总结：
针对多模态大型语言模型在第一视角视频理解中易产生幻觉的问题，提出首个基准EgoIllusion（含1400段视频、8000个问题），评估发现强模型如GPT-4o准确率仅59%，为低幻觉率第一视角MLLMs开发奠定基础。
摘要：
多模态大型语言模型（MLLMs）在复杂多模态任务中表现出色，虽在第三人称和第一视角视频的视觉感知与推理上表现优异，但易产生幻觉，生成连贯却不准确的响应。我们提出EgoIllusion，首个评估MLLMs在第一视角视频中幻觉问题的基准。该基准包含1400段视频及8000个人工标注的开放式与封闭式问题，旨在触发第一视角视频中视觉和听觉线索的幻觉。对10个MLLMs的评估显示存在显著挑战，包括GPT-4o和Gemini等强模型仅达59%准确率。EgoIllusion为开发鲁棒基准以评估MLLMs效果奠定基础，推动低幻觉率第一视角MLLMs的发展，且将开源以确保可复现性。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in complex multimodal tasks. While MLLMs excel at visual perception and reasoning in third-person and egocentric videos, they are prone to hallucinations, generating coherent yet inaccurate responses. We present EgoIllusion, a first benchmark to evaluate MLLM hallucinations in egocentric videos. EgoIllusion comprises 1,400 videos paired with 8,000 human-annotated open and closed-ended questions designed to trigger hallucinations in both visual and auditory cues in egocentric videos. Evaluations across ten MLLMs reveal significant challenges, including powerful models like GPT-4o and Gemini, achieving only 59% accuracy. EgoIllusion lays the foundation in developing robust benchmarks to evaluate the effectiveness of MLLMs and spurs the development of better egocentric MLLMs with reduced hallucination rates. Our benchmark will be open-sourced for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
