Evaluation of Audio Language Models for Fairness, Safety, and Security
Computer Science, Taibah University, Saudi Arabia(塔布大学计算机科学系,沙特阿拉伯);La Sapienza, University of Rome, Rome, Italy(罗马大学拉·萨皮恩扎分校,意大利);CISPA Helmholtz Center for Information Security, Saarbrücken, Germany(信息安全赫尔姆霍茨研究中心,德国萨尔布吕肯);University of Cagliari, Cagliari, Italy(卡利亚里大学,意大利卡利亚里)
Audio large language models (ALLMs) have recently advanced spoken interaction by integrating speech processing with large language models. However, existing evaluations of fairness, safety, and security (FSS) remain fragmented, largely because ALLMs differ fundamentally in how acoustic information is represented and where semantic reasoning occurs. Differences that are rarely made explicit. As a result, evaluations often conflate structurally distinct systems, obscuring the relationship between model design and observed FSS behavior. In this work, we introduce a structural taxonomy (system-level and representational) of ALLMs that categorizes systems along two axes: the form of audio input representation (e.g., discrete vs. continuous) and the locus of semantic reasoning (e.g., cascaded, multimodal, or audio-native). Building on the taxonomy, we propose a unified evaluation framework that assesses semantic invariance under paralinguistic variation, refusal and toxicity behavior under unsafe prompts, and robustness to adversarial audio perturbations. We apply this framework to two representative systems and observe systematic differences in refusal rates, attack success, and toxicity between audio and text inputs. Our findings demonstrate that FSS behavior is tightly coupled to how acoustic information is integrated into semantic reasoning, underscoring the need for structure-aware evaluation of audio language models.
From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs
Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Nanyang Technological University(南洋理工大学);JIUTIAN Research(中移九天);Tencent ARC Lab(腾讯ARC实验室);Chongqing University of Posts and Telecommunications(重庆邮电大学)
InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?
Centre for Responsible AI, Indian Institute of Technology Madras(印度马德拉斯理工学院负责任人工智能中心);International Institute of Information Technology, Hyderabad(海德拉巴国际信息技术学院);AmexAI Labs, American Express(美国运通AmexAI实验室);Wadhwani School of Data Science and AI, Indian Institute of Technology Madras(印度马德拉斯理工学院瓦德瓦尼数据科学与人工智能学院)
JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));State Key Laboratory of Internet Architecture, Tsinghua University(清华大学互联网架构国家重点实验室);University of North Texas(北卡罗来纳州立大学);University of Science and Technology of China(中国科学技术大学);Fujian Normal University(福建师范大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学)
VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan(台湾大学通信工程研究所);NVIDIA, Taiwan(台湾NVIDIA);Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan(台湾大学人工智能卓越研究中心)
ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);National University of Singapore(新加坡国立大学);CSIRO’s Data61(CSIRO数据61);Responsible AI Research (RAIR) Centre, The University of Adelaide(负责任人工智能研究(RAIR)中心,阿德莱德大学);Tsinghua University(清华大学)
PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations using LLMs with Enhanced Safety, Fairness, and Controllability
Posts and Telecommunications Institute of Technology(邮电技术研究所);Hanoi Architectural University(河内建筑大学)
Acoustic Interference: A New Paradigm Weaponizing Acoustic Latent Semantic for Universal Jailbreak against Large Audio Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong Polytechnic University(香港理工大学);The Chinese University of Hong Kong(香港中文大学)
The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization
The State Key Laboratory of Complex and Critical Software Environment, Beihang University(复杂与关键软件环境国家重点实验室,北航);Shanghai AI Laboratory(上海人工智能实验室);The Key Laboratory of Data Science and Intelligent Computing, International Innovation Institute, Beihang University(数据科学与智能计算重点实验室,国际创新研究院,北航)
Gender Fairness in Audio Deepfake Detection: Performance and Disparity Analysis
School of Computing, Wichita State University(维斯科大学计算学院);Institut national de la recherche scientifique (INRS–EMT)(国家科学研究中心(INRS–EMT));INRS-UQO Mixed Research Unit on Cybersecurity(网络安全混合研究单位(INRS-UQO))
VocalAgent: Large Language Models for Vocal Health Diagnostics with Safety-Aware Evaluation
Massachusetts Institute of Technology, USA(麻省理工学院, 美国);Seoul National University Hospital, South Korea(首尔国立大学医院, 韩国);Doctor Diary, South Korea(Doctor Diary, 韩国);Google Research, USA(谷歌研究, 美国);Independent Researcher(独立研究者)
Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Oxford(牛津大学);Xi’an Jiaotong University(西安交通大学);Hong Kong University of Science and Technology(香港科技大学);Northeastern University(东北大学);Beijing University of Technology(北京理工大学)
Confident, Calibrated, or Complicit: Probing the Trade-offs between Safety Alignment and Ideological Bias in Language Models in Detecting Hate Speech
School of Physics, Mathematics and Computing(物理、数学与计算学系);Network Analysis and Social Influence Modelling (NASIM) Lab(网络分析与社会影响建模实验室);The University of Western Australia(西澳大学)
Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairness
Computational Modeling and Simulation University of Pittsburgh(计算建模与仿真大学匹兹堡大学);Mathematics & Statistics Department University of Minnesota Duluth(数学与统计学系明尼苏达大学 Duluth分校);Independent Researcher in AI and Statistics(人工智能与统计学独立研究者);Hardware Technology Organization(硬件技术组织)
MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Chicago(芝加哥大学);University of California, Berkeley(加州大学伯克利分校);Harvard University(哈佛大学);Massachusetts Institute of Technology(麻省理工学院);Virginia Tech(弗吉尼亚理工大学);University of Georgia(佐治亚大学);Microsoft Corporation(微软公司);Center for AI Safety(人工智能安全中心)
Towards Trustworthy Audio Deepfake Detection: A Systematic Framework for Diagnosing and Mitigating Gender Bias
School of Computing, Wichita State University(维克托利亚州立大学计算机学院);Institut national de la recherche scientifique (INRS-EMT)(国家科学研究机构(INRS-EMT));INRS-UQO Mixed Research Unit on Cybersecurity(INRS-UQO网络安全混合研究单位)