通过多样声学条件下的语音查询评估音频-视觉多模态大语言模型的幻觉
Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions
AI总结：
研究探讨语音查询对多模态幻觉的影响，通过RePOPE-Spk基准测试发现语音查询导致错误率显著增加，提出改进方法以提升语音界面可靠性。
摘要：
多模态模型中的幻觉问题已通过图像-文本查询基准进行广泛研究，但语音查询对多模态幻觉的影响仍鲜有探索。本文提出系统流程，将现有多模态幻觉基准转换为语音查询版本，保持原任务和标签。在RePOPE上实现该流程，释放RePOPE-Spk，所有查询以语音音频形式提供。实验结果表明，语音查询导致幻觉增加：清洁语音下错误率增加3-6%，环境噪声下高达30%。此外，多轮提示和推理链仅提供部分缓解。研究结果推动构建可靠语音界面系统和评估方向。
英文摘要：
Hallucinations in multimodal models have been extensively studied using benchmarks that probe reliability in image-text query settings. However, the effect of spoken queries on multimodal hallucinations remains largely unexplored, despite the growing role of voice interfaces. In this paper, we introduce a systematic pipeline that converts existing multimodal hallucination benchmarks into spoken-query versions while preserving the original tasks and labels. We instantiate this pipeline on RePOPE and release RePOPE-Spk, where all queries are provided as spoken audio under diverse input conditions. Experimental results show that hallucinations escalate when queries are spoken rather than written: error rates increase by 3-6% with clean speech and by up to 30% under environmental noise. Furthermore, many-shot prompting and chain-of-thought reasoning provide only partial mitigation. Our findings motivate new directions for building reliable voice interface systems and evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
