arXiv 2510.104442025-10-17cs.CLcs.AI

音频大语言模型真的在“听”吗，还是仅仅在转录？衡量词汇与声学情感线索依赖

Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance

Department of Linguistics, The Ohio State University, USA（语言学系，俄亥俄州立大学）；Department of Computer Science and Engineering, The Ohio State University, USA（计算机科学与工程系，俄亥俄州立大学）；Department of Information Sciences and Technology, Penn State University, USA（信息科学与技术系，宾夕法尼亚州立大学）；Amazon, USA（亚马逊公司）

Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, Micha Elsner

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出LISTEN基准，评估六个音频大语言模型，发现它们主要依赖词汇而非声学线索，在情感理解上表现为“转录”而非“倾听”，并提供了评估框架。

摘要：

从语音中理解情感需要对词汇和声学线索都保持敏感。然而，目前尚不清楚大型音频语言模型（LALMs）是真正处理声学信息，还是主要依赖词汇内容。我们提出了LISTEN（叙事中情感的词汇与声学语音测试），这是一个受控基准，旨在区分情感理解中的词汇依赖与声学敏感性。在对六个最先进的LALM进行评估时，我们观察到一致的词汇主导现象。当词汇线索为中性或缺失时，模型预测“中性”；在线索一致时，模型性能提升有限；在线索冲突时，模型无法对不同的情感进行分类。在副语言环境中，性能接近随机水平。这些结果表明，当前的LALM在很大程度上是“转录”而非“倾听”，它们严重依赖词汇语义，而对声学线索利用不足。LISTEN为评估多模态模型中的情感理解提供了一个原则性框架。

英文摘要：

Understanding emotion from speech requires sensitivity to both lexical and acoustic cues. However, it remains unclear whether large audio language models (LALMs) genuinely process acoustic information or rely primarily on lexical content. We present LISTEN (Lexical vs. Acoustic Speech Test for Emotion in Narratives), a controlled benchmark designed to disentangle lexical reliance from acoustic sensitivity in emotion understanding. Across evaluations of six state-of-the-art LALMs, we observe a consistent lexical dominance. Models predict "neutral" when lexical cues are neutral or absent, show limited gains under cue alignment, and fail to classify distinct emotions under cue conflict. In paralinguistic settings, performance approaches chance. These results indicate that current LALMs largely "transcribe" rather than "listen," relying heavily on lexical semantics while underutilizing acoustic cues. LISTEN offers a principled framework for assessing emotion understanding in multimodal models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。