通过音频感知解码减少大型音频语言模型中的对象幻觉
Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding
AI总结：
本文提出了一种轻量级推理时策略Audio-Aware Decoding（AAD），通过对比有/无音频上下文时的词元预测logits来提升音频相关词元概率，有效缓解了大型音频语言模型的对象幻觉问题，并在对象幻觉和通用音频问答数据集上显著提升了性能。
摘要：
大型音频语言模型（LALMs）可以接收音频和文本作为输入，并回答关于音频的问题。尽管现有的LALMs在标准基准上表现出色，但有令人担忧的证据表明，LALMs会对音频中呈现的内容产生幻觉。为了缓解LALMs的幻觉问题，我们引入了Audio-Aware Decoding（AAD），这是一种轻量级的推理时策略，它使用对比解码来比较有和没有音频上下文时的词元预测logits。通过对比解码，AAD提升了那些在音频存在时概率增加的词元。我们在对象幻觉数据集上使用三个LALMs进行了实验，结果表明AAD将F1分数提高了0.046到0.428。我们还表明，AAD可以在Clotho-AQA等通用音频问答数据集上将准确率提高5.4%到10.3%。我们进行了彻底的消融研究，以了解AAD中每个组件的有效性。
英文摘要：
Large Audio-Language Models (LALMs) can take audio and text as the inputs and answer questions about the audio. While prior LALMs have shown strong performance on standard benchmarks, there has been alarming evidence that LALMs can hallucinate what is presented in the audio. To mitigate the hallucination of LALMs, we introduce Audio-Aware Decoding (AAD), a lightweight inference-time strategy that uses contrastive decoding to compare the token prediction logits with and without the audio context. By contrastive decoding, AAD promotes the tokens whose probability increases when the audio is present. We conduct our experiment on object hallucination datasets with three LALMs and show that AAD improves the F1 score by 0.046 to 0.428. We also show that AAD can improve the accuracy on general audio QA datasets like Clotho-AQA by 5.4% to 10.3%. We conduct thorough ablation studies to understand the effectiveness of each component in AAD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
