AHA: 通过反事实硬负样本对齐大音频-语言模型以缓解推理幻觉
AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives
AI总结：
AHA通过反事实硬负样本对齐大音频-语言模型，有效减少推理幻觉，提升时间推理能力，并在多个基准测试中取得显著成效。
摘要：
尽管大型音频-语言模型（LALMs）在性能上达到最先进的水平，但它们经常会出现幻觉，例如生成与音频输入无关的文本。我们分析了这些基础失败并确定了不同的分类：事件遗漏、错误事件身份、时间关系错误以及定量时间错误。为了解决这个问题，我们引入了AHA（音频幻觉对齐）框架。通过利用反事实硬负样本挖掘，我们的流程构建了一个高质量的偏好数据集，迫使模型区分严格的声学证据与语言上合理的虚构。此外，我们建立了AHA-Eval，一个用于严格测试这些细粒度时间推理能力的诊断基准。我们将此数据应用于对齐Qwen2.5-Omni。结果模型Qwen-Audio-AHA在AHA-Eval上实现了13.7%的提升。至关重要的是，这种收益超越了我们的诊断集。我们的模型在公开基准上显示出显著的改进，包括在MMAU-Test上提高1.3%和在MMAR上提高1.6%，优于最新最先进的方法。模型和数据集在https://github.com/LLM-VLM-GSL/AHA上开源。
英文摘要：
Although Large Audio-Language Models (LALMs) deliver state-of-the-art (SOTA) performance, they frequently suffer from hallucinations, e.g. generating text not grounded in the audio input. We analyze these grounding failures and identify a distinct taxonomy: Event Omission, False Event Identity, Temporal Relation Error, and Quantitative Temporal Error. To address this, we introduce the AHA (Audio Hallucination Alignment) framework. By leveraging counterfactual hard negative mining, our pipeline constructs a high-quality preference dataset that forces models to distinguish strict acoustic evidence from linguistically plausible fabrications. Additionally, we establish AHA-Eval, a diagnostic benchmark designed to rigorously test these fine-grained temporal reasoning capabilities. We apply this data to align Qwen2.5-Omni. The resulting model, Qwen-Audio-AHA, achieves a 13.7% improvement on AHA-Eval. Crucially, this benefit generalizes beyond our diagnostic set. Our model shows substantial gains on public benchmarks, including 1.3% on MMAU-Test and 1.6% on MMAR, outperforming latest SOTA methods. The model and dataset are open-sourced at https://github.com/LLM-VLM-GSL/AHA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
