arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.09321cs.CVcs.AIcs.LG
收藏

DAVE: 音频视觉评估诊断基准

DAVE: Diagnostic benchmark for Audio Visual Evaluation

KU Leuven(卢森堡大学)

Gorjan Radevski, Teodora Popordanoska, Matthew B. Blaschko, Tinne Tuytelaars

AI总结:

DAVE提出一个诊断基准,通过确保两种模态必要性及分解评估子类,解决多模态模型评估中的视觉偏见问题,提供更精准的模型诊断与改进指导。

摘要:

音频视觉理解是一个快速发展的领域,旨在整合和解释来自听觉和视觉模态的信息。尽管多模态学习近年来取得了进展,但现有基准往往存在强烈的视觉偏见——当答案可以仅从视觉数据中推断出来时——并且只提供汇总分数,将多种误差源混合在一起。这使得难以确定模型是否在视觉理解、音频解释或音频-视觉对齐方面遇到困难。在本工作中,我们引入了DAVE:诊断音频视觉评估,一个全新的基准数据集,旨在系统地在受控环境中评估音频-视觉模型。DAVE通过(i)确保两种模态都必要才能正确回答,以及(ii)将评估分解为原子子类别来缓解现有局限性。我们对最新模型的详细分析揭示了特定的失败模式,并提供了改进的针对性见解。通过提供这种标准化的诊断框架,我们旨在促进更稳健的音频-视觉模型发展。

英文摘要:

Audio-visual understanding is a rapidly evolving field that seeks to integrate and interpret information from both auditory and visual modalities. Despite recent advances in multi-modal learning, existing benchmarks often suffer from strong visual bias -- when answers can be inferred from visual data alone -- and provide only aggregate scores that conflate multiple sources of error. This makes it difficult to determine whether models struggle with visual understanding, audio interpretation, or audio-visual alignment. In this work, we introduce DAVE: Diagnostic Audio Visual Evaluation, a novel benchmark dataset designed to systematically evaluate audio-visual models across controlled settings. DAVE alleviates existing limitations by (i) ensuring both modalities are necessary to answer correctly and (ii) decoupling evaluation into atomic subcategories. Our detailed analysis of state-of-the-art models reveals specific failure modes and provides targeted insights for improvement. By offering this standardized diagnostic framework, we aim to facilitate more robust development of audio-visual models. Dataset: https://huggingface.co/datasets/gorjanradevski/dave Code: https://github.com/gorjanradevski/dave

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑