arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16403cs.CVcs.SD
收藏

当视觉为声音说话

When Vision Speaks for Sound

University of California, Davis(加州大学戴维斯分校)

Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai, Tinghui Zhu, Wendi Li, Yanan Xie, Muhao Chen, Peng Qi

AI总结:

本文发现视频中MLLMs的音频理解依赖视觉线索而非实际音频流,提出Thud框架通过三种音频编辑干预研究此问题,并提出两阶段对齐方法提升模型性能。

摘要:

尽管视频能力的MLLMs取得显著进展,我们发现其视频中的音频理解往往由视觉驱动:模型依赖视觉线索推断或虚构音频信息,而非验证音频流。此问题出现在最先进的开源全能模型和领先的闭源模型中。我们将此失败模式称为音频-视觉的Clever Hans效应,即模型看似(错误地)音频相关,但实际利用视觉-音频相关性而不验证音频和视觉流是否真正对齐。为系统研究此行为,我们引入Thud,一个基于三种反事实音频编辑的干预驱动探测框架:Shift测试时间同步,Mute测试声音存在,Swap测试音频-视觉一致性。除诊断外,我们进一步研究两阶段对齐方法:干预衍生的偏好对教授音频验证,而事件级通用视频偏好规范模型防止过度专业化。我们的最佳10000样本方法在三个干预维度的平均性能提高28个百分点,同时略微提升通用视频和音频-视觉问答基准性能。

英文摘要:

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑