Consider a scenario where a harmfulness evaluation metric intended to filter unsafe responses from a Large Language Model. When applied to individual harmful prompt-response pairs, it correctly flags them as unsafe by assigning a high-risk score. Yet, if those same pairs are concatenated, the metrics decision unexpectedly reverses - labelling the combined content as safe with a low score, allowing the harmful text to bypass the filter. We found that multiple safety metrics, including advanced metrics such as GPT-based judges, exhibit this non-safe behaviour. Moreover, they show a strong sensitivity to input order: responses are often classified as safe if safe content appears first, regardless of any harmful content that follows, and vice versa. These findings underscore the importance of evaluating the safety of safety metrics, that is, the reliability of their output scores. To address this, we developed general, automatic, concatenation-based tests to assess key properties of these metrics. When applied in a model safety scenario, the tests revealed significant inconsistencies in harmfulness evaluations.
A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；King Abdullah University of Science and Technology（卡塔尔国王大学科学与技术研究院）；University of Oxford（牛津大学）
Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models
Shanghai Key Laboratory of Computer Software Testing and Evaluating（上海软件测试与评估 key laboratory）；Shanghai Normal University（上海Normal University）；TrustAI Pte. Ltd.；East China Normal University（东华师范大学）
Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）；Sun Yat-sen University（中山大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Safety Aware Task Planning via Large Language Models in Robotics
University of Minnesota – Twin Cities（明尼苏达大学双城分校）；Institute for Robotics and Intelligent Machines (IRIM), Georgia Institute of Technology（佐治亚理工学院机器人与智能机器研究所）；Pontificia Universidad Católica de Chile（智利天主教大学）
Multilingual Safety Signals Are Multi-Layered: Filtering Safety-Degrading Data for Safer LLMs
Beihang University（北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）；Qingdao Research Institute, Beihang University（北京航空航天大学青岛研究院）
Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
Department of Computer Science and Technology, BNRist（计算机科学与技术系，BNRist）；Shenzhen International Graduate School（深圳国际研究生院）；KIRC, Institute for Artificial Intelligence, Tsinghua University（人工智能研究院，清华大学）
Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios
Zhongguancun Laboratory（中关村实验室）；Beihang University（北京航空航天大学）；Institute of Data Space, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；A*STAR（新加坡科技研究局）
A Safety Modulator Actor-Critic Method in Model-Free Safe Reinforcement Learning and Application in UAV Hovering
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息工程学院）；City University of Hong Kong（香港城市大学）；Institute of Unmanned Aerial Vehicle Systems, Lingchuan Industry（凌川工业无人机系统研究所）
Predictive Safety Shield for Dyna-Q Reinforcement Learning
Departement U2IS, ENSTA Paris, Institut Polytechnique de Paris（ENSTA巴黎大学U2IS部门，巴黎理工学院）；Department of Electrical Engineering and Computer Science, University of California, Berkeley, USA（加州大学伯克利分校电气工程与计算机科学系）
