VoxSafeBench:不仅仅是所说的内容,还有谁、如何以及在哪里
VoxSafeBench: Not Just What Is Said, but Who, How, and Where
AI总结:
VoxSafeBench首次联合评估SLM在安全、公平和隐私三个维度上的社会契合度,揭示语音识别中的普遍缺口,即模型在文本中能识别社会规范,但在语音中却无法正确应用。
摘要:
随着语音语言模型(SLMs)从个人设备转向共享、多用户环境,其响应必须考虑的不仅限于词语本身。说话者是谁、声音如何以及对话发生的地点,都可能使原本无害的请求变成不安全、不公正或侵犯隐私的请求。然而,现有基准主要关注基本语音理解,研究个体风险或混淆本质上有害的内容与因声音环境而变得有问题的内容。我们引入VoxSafeBench,是首个联合评估SLM在安全、公平和隐私三个维度上的社会契合度的基准。VoxSafeBench采用双层设计:第一层使用匹配的文本和音频输入评估内容相关风险,第二层针对音频条件化风险,即文本无害但适当响应取决于说话者、副语言线索或周围环境。为了验证第二层,我们包含中间感知探测器并确认前沿SLMs能够成功检测这些声音线索但仍无法适当应对。在22个任务中,我们发现看似在文本上稳健的防护措施在语音中会退化:安全意识下降对于说话者和场景条件化风险,公平性在语音中传达人口差异时会削弱,隐私保护在声音线索到来时会崩溃。这些结果揭示了语音接地的普遍缺口:当前SLMs经常在文本中识别相关社会规范,但在决定性线索必须基于语音时却无法正确应用。代码和数据可在:https://amphionteam.github.io/VoxSafeBench_demopage/
英文摘要:
As speech language models (SLMs) transition from personal devices into shared, multi-user environments, their responses must account for far more than the words alone. Who is speaking, how they sound, and where the conversation takes place can each turn an otherwise benign request into one that is unsafe, unfair, or privacy-violating. Existing benchmarks, however, largely focus on basic audio comprehension, study individual risks in isolation, or conflate content that is inherently harmful with content that only becomes problematic due to its acoustic context. We introduce VoxSafeBench, among the first benchmarks to jointly evaluate social alignment in SLMs across three dimensions: safety, fairness, and privacy. VoxSafeBench adopts a Two-Tier design: Tier1 evaluates content-centric risks using matched text and audio inputs, while Tier2 targets audio-conditioned risks in which the transcript is benign but the appropriate response hinges on the speaker, paralinguistic cues, or the surrounding environment. To validate Tier2, we include intermediate perception probes and confirm that frontier SLMs can successfully detect these acoustic cues yet still fail to act on them appropriately. Across 22 tasks with bilingual coverage, we find that safeguards appearing robust on text often degrade in speech: safety awareness drops for speaker- and scene-conditioned risks, fairness erodes when demographic differences are conveyed vocally, and privacy protections falter when contextual cues arrive acoustically. Together, these results expose a pervasive speech grounding gap: current SLMs frequently recognize the relevant social norm in text but fail to apply it when the decisive cue must be grounded in speech. Code and data are publicly available at: https://amphionteam.github.io/VoxSafeBench_demopage/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。