SVHalluc: 音频-视觉大语言模型中的语音-视觉幻觉基准测试
SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models
AI总结：
针对音频-视觉大语言模型中的语音-视觉幻觉问题，提出SVHalluc基准，从语义和时间两个维度评估模型将语音内容与视觉信号对齐的能力，发现现有模型存在跨模态理解局限。
摘要：
尽管音频-视觉大语言模型（LLMs）取得了成功，但它们可能产生看似合理但缺乏依据的输出，即幻觉。现有基准侧重于环境声音（例如狗叫）来指示事件发生。相比之下，人类语音承载着根本不同的、丰富的语义和时间结构，但当前模型能否准确地将语音内容与相应的视觉信号对齐仍未得到探索。在这项工作中，我们表明语音内容可以引发音频-视觉LLMs中的幻觉。为了系统研究这一点，我们引入了SVHalluc，这是第一个用于评估音频-视觉LLMs中语音-视觉幻觉的综合基准。我们的基准从两个关键且互补的方面诊断语音-视觉幻觉：语义和时间。实验结果表明，最先进的开源音频-视觉LLMs难以将语音内容与相应的视觉信号对齐，在多个任务上的准确率接近随机。相比之下，Gemini 2.5 Pro显著优于开源模型。我们的分析表明，它们的失败源于跨模态理解能力有限，尽管在单模态感知方面表现强劲。我们的工作揭示了当前音频-视觉LLMs的一个新的根本性局限，并强调了基于语音的视频理解的需求。项目页面：此https URL。
英文摘要：
Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination. Existing benchmarks focus on environmental sounds (e.g., dog barking) to indicate event occurrence. In contrast, human speech carries fundamentally different, rich semantics and temporal structures, yet it remains unexplored whether current models can accurately align speech content with corresponding visual signals. In this work, we show that speech content can induce hallucinations in audio-visual LLMs. To systematically study this, we introduce SVHalluc, the first comprehensive benchmark for evaluating speech-vision hallucination in audio-visual LLMs. Our benchmark diagnoses speech-vision hallucinations from two critical and complementary aspects: semantic and temporal. Experimental results demonstrate that state-of-the-art open-source audio-visual LLMs struggle with aligning speech content with corresponding visual signals, with a near-random accuracy on multiple tasks. In contrast, Gemini 2.5 Pro significantly outperforms the open-source models. Our analysis suggests that their failures stem from limited ability in cross-modality understanding, despite strong performance in single-modality perception. Our work uncovers a new and fundamental limitation of current audio-visual LLMs and highlights the need for speech-grounded video comprehension. Project page: https://chenshuang-zhang.github.io/projects/svhalluc/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
