当音频大模型不听：一种跨语言的模态仲裁研究
When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration
AI总结：
研究发现音频与文本冲突时，语言模型更倾向于遵循文本而非音频，通过ALME数据集和TDR指标揭示模态仲裁中的信息内容与仲裁易用性差异。
摘要：
当音频和文本冲突时，语音启用的语言模型在仲裁两个冲突文本源时，比在仲裁音频与文本时更频繁地遵循文本，即使在明确指示信任音频的情况下也是如此。我们介绍了包含57,602个跨八种语言受控音频-文本冲突刺激的ALME数据集，以及文本主导率（TDR），该指标衡量模型在被指示遵循音频时遵循冲突文本的频率。Gemini 2.0 Flash和GPT-4o的TDR比在其他条件下将音频替换为其转录本的基线模型高10-26倍（Gemini 2.0 Flash: 16.6% vs. 1.6%；GPT-4o: 23.2% vs. 0.9%）。这些结果表明，文本主导不仅反映了信息内容，还反映了仲裁可访问性的不对称性，即模型在决策时使用竞争性表示的难易程度。将转录本框架视为故意损坏会减少TDR 80%，而强制显式转录会增加TDR 14%。进一步的微调消融分析表明，仲裁行为更多取决于LLM推理而非音频输入路径本身。在四个音频-大语言模型中，我们观察到相同的定性模式，但存在显著的跨模型和跨语言变化。
英文摘要：
When audio and text conflict, speech-enabled language models follow text far more often than they do when arbitrating between two conflicting text sources, even under explicit instructions to trust the audio. We introduce ALME (Audio-LLM Modality Evaluation), a dataset of 57,602 controlled audio-text conflict stimuli across eight languages, together with Text Dominance Ratio (TDR), which measures how often a model follows conflicting text when instructed to follow audio. Gemini 2.0 Flash and GPT-4o show TDR 10--26$\times$ higher than a baseline that replaces audio with its transcript under otherwise identical conditions (Gemini 2.0 Flash: 16.6% vs. 1.6%; GPT-4o: 23.2% vs. 0.9%). These results suggest that text dominance reflects not only information content, but also an asymmetry in arbitration accessibility, i.e., how easily the model can use competing representations at decision time. Framing the transcript as deliberately corrupted reduces TDR by 80%, whereas forcing explicit transcription increases it by 14%. A fine-tuning ablation further suggests that arbitration behavior depends more on LLM reasoning than on the audio input path alone. Across four audio-LLMs, we observe the same qualitative pattern with substantial cross-model and cross-linguistic variation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
