arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04796eess.AScs.SD
收藏

LALM-as-a-Judge:用于多轮口语对话安全评估的大型音频语言模型基准测试

LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues

Computer Engineering, Technion--Israel Institute of Technology, Haifa, Israel(技术学院电子工程系,技术离子技术研究所,以色列海法);Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA(语言技术研究所,卡内基梅隆大学,美国匹兹堡)

Amir Ivry, Shinji Watanabe

AI总结:

针对口语对话中社会不安全内容评估仍以文本为中心、忽略韵律和转录失败的问题,提出包含24000个多轮口语对话的开放基准,评估6种大型音频语言模型在文本、音频和多模态设置下的敏感性、严重性顺序特异性和轮次位置偏差,发现音频提供非词汇证据,多模态增益非普遍且存在多种模式。

摘要:

对口语对话中社会不安全内容的评估仍然以文本为中心,忽略了韵律和转录失败。我们提出了LALM-as-a-Judge,其中包括一个包含24000个多轮口语对话的开放基准,每个对话包含一个局部不安全轮次,这些对话基于8个社会不安全类别和5个严重级别生成。我们评估了6种大型音频语言模型(LALMs)作为评判者,包括开源和闭源模型,在纯文本、纯音频和多模态设置下,针对对话中社会有害内容的敏感性、严重性顺序特异性和轮次位置偏差。结果表明,音频提供了超越转录语义的非词汇证据,并且多模态增益并非普遍存在,而是可以表现为文本锚定、平衡、保守和干扰,我们将这些归因于音频路径瓶颈和融合限制。我们将该基准定位为诊断工具,并为模型、模态和提示选择提供实践者指导。

英文摘要:

Evaluation of socially unsafe content in spoken dialogues remains text-centric, missing prosody and transcription failures. We present LALM-as-a-Judge, which includes an open benchmark of 24,000 multi-turn spoken dialogues with one localized unsafe turn, generated out of 8 socially unsafe categories and 5 severity levels. We evaluate 6 large audio-language models (LALMs) as judges, open and closed-source, in text-only, audio-only, and multimodal setups by their sensitivity, severity-order specificity, and turn-position bias for socially harmful content in the dialogue. Results show that audio contributes non-lexical evidence beyond transcript semantics and that multimodal gains are not universal but can be text-anchored, balanced, conservative, and interfering, which we link to the audio pathway bottlenecks and fusion limits. We position the benchmark as diagnostic and derive practitioner guidance for model, modality, and prompts choices.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑