检测并缓解视频到音频生成中的插入幻觉
Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation
AI总结：
本文提出HALCON方法，通过三阶段流程缓解视频到音频生成中的插入幻觉问题，显著降低幻觉的普遍性和持续时间，同时保持音频质量和时间同步性能。
摘要：
视频到音频生成在自动合成视频声音方面取得了显著进展。然而，现有的评估指标，专注于语义和时间对齐，忽略了关键的失败模式：模型经常生成没有对应视觉来源的音频事件，特别是语音和音乐。我们将这种现象称为插入幻觉，并将其识别为由数据集偏见驱动的系统性风险，例如屏幕外声音的普遍性，这种现象目前完全未被现有指标检测到。为了解决这一挑战，我们首先开发了一个系统性的评估框架，该框架使用多个音频事件检测器的多数投票集成。我们还引入了两个新的指标来量化该问题的普遍性和严重性：IH@vid（具有幻觉的视频比例）和IH@dur（幻觉持续时间比例）。基于此，我们引入HALCON来缓解IH。HALCON遵循三阶段过程：首先生成初始音频以暴露幻觉段，然后识别并掩码相应的不可靠视频特征，最后使用纠正的条件重新生成音频。在几个主流V2A基准上的实验首先揭示了当前最先进的模型存在严重的IH。相比之下，我们的HALCON方法在平均上将幻觉的普遍性和持续时间减少了超过50%，同时不降低，甚至在某些情况下提高了传统音频质量和时间同步的指标。我们的工作是首次正式定义、系统测量并有效缓解插入幻觉的研究，为更可靠和忠实的V2A模型铺平了道路。
英文摘要：
Video-to-Audio generation has made remarkable strides in automatically synthesizing sound for video. However, existing evaluation metrics, which focus on semantic and temporal alignment, overlook a critical failure mode: models often generate acoustic events, particularly speech and music, that have no corresponding visual source. We term this phenomenon Insertion Hallucination and identify it as a systemic risk driven by dataset biases, such as the prevalence of off-screen sounds, that remains completely undetected by current metrics. To address this challenge, we first develop a systematic evaluation framework that employs a majority-voting ensemble of multiple audio event detectors. We also introduce two novel metrics to quantify the prevalence and severity of this issue: IH@vid (the fraction of videos with hallucinations) and IH@dur (the fraction of hallucinated duration). Building on this, we introduce HALCON to mitigate IH. HALCON follows a three-stage procedure: it first generates initial audio to expose hallucinated segments, then identifies and masks the corresponding unreliable video features, and finally regenerates the audio using the corrected conditioning. Experiments on several mainstream V2A benchmarks first reveal that state-of-the-art models suffer from severe IH. In contrast, our HALCON method reduces both the prevalence and duration of hallucinations by over 50\% on average, without degrading, and in some cases even improving, conventional metrics for audio quality and temporal synchronization. Our work is the first to formally define, systematically measure, and effectively mitigate Insertion Hallucination, paving the way for more reliable and faithful V2A models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
