倾听者之偏见：评估音频语言模型在语言、人口统计和位置变化中的敏感性
Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations
AI总结：
本研究通过BiasInEar数据集评估音频语言模型在语言、人口统计和位置变化中的敏感性，揭示语音放大结构性偏见的机制，并提出统一的公平性与稳健性评估框架。
摘要：
本研究首次系统地调查了多语言大语言模型中的语音偏见。我们构建并发布了BiasInEar数据集，这是一个基于Global MMLU Lite的语音增强基准，涵盖英语、中文和韩语，按性别和口音平衡，总时长70.8小时（约4,249分钟），包含11,200个问题。使用四个互补指标（准确性、熵、APES和Fleiss的κ），我们在语言（语言和口音）、人口统计（性别）和结构（选项顺序）扰动下评估了九个代表性模型。我们的发现表明，MLLMs对人口因素相对稳健，但对语言和选项顺序高度敏感，表明语音可以放大现有的结构性偏见。此外，架构设计和推理策略显著影响跨语言的稳健性。总体而言，本研究建立了一个统一的框架，用于评估集成语音的LLM的公平性和稳健性，弥合了文本和语音评估之间的差距。资源可在https://github.com/ntunlplab/BiasInEar找到。
英文摘要：
This work presents the first systematic investigation of speech bias in multilingual MLLMs. We construct and release the BiasInEar dataset, a speech-augmented benchmark based on Global MMLU Lite, spanning English, Chinese, and Korean, balanced by gender and accent, and totaling 70.8 hours ($\approx$4,249 minutes) of speech with 11,200 questions. Using four complementary metrics (accuracy, entropy, APES, and Fleiss' $κ$), we evaluate nine representative models under linguistic (language and accent), demographic (gender), and structural (option order) perturbations. Our findings reveal that MLLMs are relatively robust to demographic factors but highly sensitive to language and option order, suggesting that speech can amplify existing structural biases. Moreover, architectural design and reasoning strategy substantially affect robustness across languages. Overall, this study establishes a unified framework for assessing fairness and robustness in speech-integrated LLMs, bridging the gap between text- and speech-based evaluation. The resources can be found at https://github.com/ntunlplab/BiasInEar.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
