arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.13262cs.SDcs.AI
收藏

音频语言模型的公平性、安全性和安全性评估

Evaluation of Audio Language Models for Fairness, Safety, and Security

Computer Science, Taibah University, Saudi Arabia(塔布大学计算机科学系,沙特阿拉伯);La Sapienza, University of Rome, Rome, Italy(罗马大学拉·萨皮恩扎分校,意大利);CISPA Helmholtz Center for Information Security, Saarbrücken, Germany(信息安全赫尔姆霍茨研究中心,德国萨尔布吕肯);University of Cagliari, Cagliari, Italy(卡利亚里大学,意大利卡利亚里)

Ranya Aloufi, Srishti Gupta, Soumya Shaw, Battista Biggio, Lea Schönherr

AI总结:

本文提出一种结构化分类方法,评估音频语言模型在公平性、安全性和安全性方面的表现,揭示音频信息整合对语义推理的影响。

摘要:

音频大语言模型(ALLMs)通过整合语音处理与大语言模型,最近在语音交互中取得了进展。然而,现有公平性、安全性和安全性(FSS)评估仍碎片化,主要因为ALLMs在如何表示语音信息和语义推理位置上存在根本差异。这些差异很少被明确说明。因此,评估往往将结构上不同的系统混为一谈,模糊了模型设计与观察到的FSS行为之间的关系。在本文中,我们引入了ALLMs的结构分类法(系统级和表示级),根据两个轴对系统进行分类:音频输入表示的形式(例如离散与连续)和语义推理的位置(例如级联、多模态或音频原生)。基于此分类法,我们提出了一种统一的评估框架,评估语义不变性在语调变化下的表现、在不安全提示下的拒绝和毒性行为以及对对抗性音频扰动的鲁棒性。我们应用此框架到两个代表性系统,并观察到音频和文本输入之间在拒绝率、攻击成功率和毒性方面存在系统性差异。我们的发现表明,FSS行为与如何将语音信息整合到语义推理中紧密相关,强调了对音频语言模型进行结构感知评估的必要性。

英文摘要:

Audio large language models (ALLMs) have recently advanced spoken interaction by integrating speech processing with large language models. However, existing evaluations of fairness, safety, and security (FSS) remain fragmented, largely because ALLMs differ fundamentally in how acoustic information is represented and where semantic reasoning occurs. Differences that are rarely made explicit. As a result, evaluations often conflate structurally distinct systems, obscuring the relationship between model design and observed FSS behavior. In this work, we introduce a structural taxonomy (system-level and representational) of ALLMs that categorizes systems along two axes: the form of audio input representation (e.g., discrete vs. continuous) and the locus of semantic reasoning (e.g., cascaded, multimodal, or audio-native). Building on the taxonomy, we propose a unified evaluation framework that assesses semantic invariance under paralinguistic variation, refusal and toxicity behavior under unsafe prompts, and robustness to adversarial audio perturbations. We apply this framework to two representative systems and observe systematic differences in refusal rates, attack success, and toxicity between audio and text inputs. Our findings demonstrate that FSS behavior is tightly coupled to how acoustic information is integrated into semantic reasoning, underscoring the need for structure-aware evaluation of audio language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑