探索第一人称视频理解中的音频幻觉
Exploring Audio Hallucination in Egocentric Video Understanding
AI总结：
本文研究了第一人称视频中音频幻觉问题，提出自动评估框架和分类体系，发现先进AV-LLMs在回答声音相关问题时存在较高幻觉率。
摘要：
第一人称视频提供了一个独特场景，在此声音是理解用户活动和环境的关键线索，特别是在视觉信息不稳定或被遮挡时。最先进的大型音频视觉语言模型（AV-LLMs）能够生成多模态描述。然而，本文表明这些模型容易产生音频幻觉，通常从可见但未听到的视觉线索中推断声音。我们提出了一种系统且自动的评估框架，通过定向问答（Q/A）协议分析第一人称视频中的音频幻觉。我们整理了300个第一人称视频数据集，并设计了1000个以声音为中心的问题来探测模型输出。为了表征幻觉，我们提出了一个扎根分类体系，区分用户活动的前景动作声音和背景环境声音。我们的评估显示，先进的AV-LLMs，如Qwen2.5 Omni，在回答与前景和背景声音相关的问题时准确率分别仅为27.3%和39.5%。通过本研究，我们强调了衡量多模态响应可靠性的必要性，强调了对幻觉的稳健评估对于开发可靠的AV-LLMs至关重要。
英文摘要：
Egocentric videos provide a distinctive setting in which sound serves as crucial cues to understand user activities and surroundings, particularly when visual information is unstable or occluded due to continuous camera movement. State-of-the-art large audio-visual language models (AV-LLMs) can generate multimodal descriptions. However, we show in this work that they are prone to audio hallucinations, often inferring sounds from visual cues that are visible but not heard. We present a systematic and automatic evaluation framework for analyzing audio hallucinations in egocentric video through a targeted question-answering (Q/A) protocol. We curate a dataset of 300 egocentric videos and design 1,000 sound-focused questions to probe model outputs. To characterize hallucinations, we propose a grounded taxonomy that distinguishes between foreground action sounds from the user activities and background ambient sounds. Our evaluation shows that advanced AV-LLMs, such as Qwen2.5 Omni, exhibit high hallucination rates, achieving only 27.3% and 39.5% accuracy on Q/As related to foreground and background sounds, respectively. With this work, we highlight the need to measure the reliability of multimodal responses, emphasizing that robust evaluation of hallucinations is essential to develop reliable AV-LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
