The rapid expansion of research in LLM safety presents challenges in tracking advancements, making benchmarks important evaluation infrastructures for identifying key trends and facilitating systematic comparisons. Yet no systematic assessment exists of their code quality and runnability, nor of what factors are associated with the community's adoption of certain benchmarks over others. To address this gap, we conduct a systematic measurement study of 31 LLM safety benchmarks (covering prompt injection, jailbreak, and hallucination) with 382 non-benchmark papers as a control group, combining automated static analysis, human runnability testing (220+ person-hours), and bibliometric analysis. We find that only 39\% of benchmark repositories can run without modification, only 16\% provide flawless installation guides, and a mere 6\% include ethical considerations despite containing potentially harmful content. These deficiencies persist across the study period with no significant improvement. Analyzing adoption factors, we find that benchmark adoption correlates with author prominence and code runnability, but not with code quality standards such as Pylint score and maintainability, suggesting that the community's benchmark selection does not reward higher coding standards. Based on these results, we identify potential safety and reliability concerns. Some safety benchmark repositories openly expose harmful content, such as successful jailbreak responses, without any ethical warning or access control, effectively serving as unguarded attack resources. Furthermore, when benchmarks require ad-hoc modifications to run, downstream safety evaluations across different papers may not be comparable. We present case studies illustrating these concrete consequences and propose a targeted checklist to help benchmark contributors improve code quality, documentation, and ethical practices.
MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers
East China Normal University(东华师范大学);Salesforce AI Research(Salesforce人工智能研究);Singapore Management University(新加坡国立大学);Shanghai AI Laboratory(上海人工智能实验室)
TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering
Critical ML Lab Waterloo Canada(Waterloo大学Critical ML实验室);FAR.AI Berkeley USA(伯克利美国FAR.AI公司);University of Toronto Toronto Canada(多伦多大学);University of Waterloo Waterloo Canada(Waterloo大学);ETH Zürich Zürich Switzerland(苏黎世联邦理工学院);MIT CSAIL Cambridge USA(麻省理工学院CSAIL实验室);University of Toronto, MPI, EuroSafeAI, Vector Institute Toronto Canada(多伦多大学、马克斯·普朗克研究所、EuroSafeAI、Vector Institute);Critical ML Lab University of Waterloo Waterloo Canada(Waterloo大学Critical ML实验室)
I'm Sorry Driver, I'm Afraid I Can't Do That: Appraising the Safety of LLMs within Automotive Contexts
UKRI AI Centre for Doctoral Training in Safe Artificial Intelligence Systems (SAINTS)(英国研究理事会安全人工智能系统博士培训中心(SAINTS));University of York(约克大学);Jaguar Land Rover(捷克·陆罗恩)
BarrierSteer: LLM Safety via Learning Barrier Steering
Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工联合研究中心);CSAIL, Massachusetts Institute of Technology(麻省理工学院计算机科学与人工智能实验室);Worcester Polytechnic Institute(沃斯堡理工学院)
Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models
School of Cyber Science and Technology, Shandong University(山东大学网络科学与技术学院);Netflix Eyeline Studios;State Key Laboratory of Cryptography and Digital Economy Security, Shandong University(山东大学密码与数字经济安全国家重点实验室);Shandong Key Laboratory of Artificial Intelligence Security, Shandong University(山东省人工智能安全重点实验室)
Secure Deep Learning Engineering: A Software Quality Assurance Perspective
Harbin Institute of Technology(哈尔滨工业大学);Nanyang Technological University(南洋理工大学);Carnegie Mellon University(卡内基梅隆大学);Optus Macquarie University Cyber Security Hub(Optus麦考瑞大学网络安全中心);Kyushu University(九州大学);University of Illinois at Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA AI Tech Centre(英伟达人工智能技术中心)
HELIOT: LLM-Based CDSS for Adverse Drug Reaction Management
Software Engineering (SeSa) Lab University of Salerno(软件工程(SeSa)实验室,萨勒诺大学);Amsterdam UMC Locatie AMC Amsterdam, The Netherlands(阿姆斯特丹大学医学中心(AMC)阿姆斯特丹,荷兰);Digital Health(数字健康;方法学;阿姆斯特丹公共卫生研究所阿姆斯特丹,荷兰);Methodology(阿姆斯特丹大学数据科学中心阿姆斯特丹,荷兰);Amsterdam Public Health Research Institute Amsterdam, The Netherlands;University of Amsterdam Data Science Center Amsterdam, The Netherlands