Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. However, subtle changes, such as shifting the order of choices, result in substantially different results. Existing MCQA frameworks do not account for this variability and report a single accuracy number per benchmark or category. We dive into the MCQA evaluation framework and conduct a systematic study spanning three benchmarks (MMAU, MMAR and MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct. Our findings indicate that models are sensitive not only to the ordering of choices, but also to the paraphrasing of the question and the choices. Finally, we propose a simpler evaluation protocol and metric that account for subtle variations and provide a more detailed evaluation report of LALMs within the MCQA framework.
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Brno University of Technology(布尔诺科技大学);Universidad Autónoma de Madrid(马德里自治大学);University of Buenos Aires(布宜诺斯艾利斯大学);Tufts University(塔夫茨大学);University of Maryland(马里兰大学)
ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood
University of Southern California(南加州大学);The Ohio State University(俄亥俄州立大学);University of California, Los Angeles(加州大学洛杉矶分校);Harvard University(哈佛大学);Boston University(波士顿大学);University of Miami(迈阿密大学)
Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Imperial College London(帝国理工学院);Technische Universität München(慕尼黑工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Shanghai Jiao Tong University(上海交通大学);Johns Hopkins University(约翰斯·霍普金斯大学)
AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan(台湾国立台湾大学通信工程研究所);Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan(台湾国立台湾大学人工智能研究中心)