arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.04584cs.CLcs.SDeess.AS
收藏

大型音频语言模型在多项选择评估中的鲁棒性评估

Robustness assessment of large audio language models in multiple-choice evaluation

Scientific Research, Telefónica Innovación Digital, Spain(Telefónica Innovación Digital科研部,西班牙);Universidad Autónoma de Madrid, Spain(马德里自治大学,西班牙);Brno University of Technology, Czech Republic(布拉格技术大学,捷克)

Fernando López, Santosh Kesiraju, Jordi Luque

AI总结:

研究大型音频语言模型在多项选择问答中对选项顺序、问题措辞的敏感性,并提出考虑细微变化的评估协议和指标。

摘要:

近年来,大型音频语言模型(LALMs)的进展主要通过多项选择问答(MCQA)框架进行评估。然而,细微的变化(如改变选项顺序)会导致结果显著不同。现有的MCQA框架未考虑这种变异性,仅报告每个基准或类别的单一准确率。我们深入研究了MCQA评估框架,并进行了系统研究,涵盖三个基准(MMAU、MMAR和MMSU)和四个模型:Audio Flamingo 2、Audio Flamingo 3、Qwen2.5-Omni-7B-Instruct和Kimi-Audio-7B-Instruct。我们的发现表明,模型不仅对选项顺序敏感,还对问题和选项的释义敏感。最后,我们提出了一个更简单的评估协议和指标,以考虑细微变化,并在MCQA框架内提供更详细的LALMs评估报告。

英文摘要:

Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. However, subtle changes, such as shifting the order of choices, result in substantially different results. Existing MCQA frameworks do not account for this variability and report a single accuracy number per benchmark or category. We dive into the MCQA evaluation framework and conduct a systematic study spanning three benchmarks (MMAU, MMAR and MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct. Our findings indicate that models are sensitive not only to the ordering of choices, but also to the paraphrasing of the question and the choices. Finally, we propose a simpler evaluation protocol and metric that account for subtle variations and provide a more detailed evaluation report of LALMs within the MCQA framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑