Achieving reproducibility, quantity, and diversity in vulnerability datasets has long been viewed as an inherent three-way trade-off, where improving one dimension often comes at the cost of the others. In practice, reproducibility has been the dimension most often neglected. This has limited what can be automatically extracted from historical bug datasets, and has reduced their utility for downstream security research.
In this work, we propose a method to produce a new security dataset which ensures reproducibility for diverse vulnerabilities at scale by identifying the key obstacles to large-scale bug reproduction and addressing them with general solutions. Using this method, we introduce full reproducibility to the largest open source software vulnerability dataset (OSS-Fuzz) and construct the ARVO dataset (an Atlas of Reproducible Vulnerabilities in Open-source software). ARVO is a large-scale dataset consisting of over 6,100 real-world vulnerabilities across 311 projects. Focusing on reproducibility, ARVO differs from existing datasets by providing each vulnerability in a form that can be consistently rebuilt, triggered, and analyzed across versions. Reproducibility also enables automatic identification of the corresponding patch for each vulnerability and supports direct interaction with vulnerabilities after code changes, capabilities that existing large-scale datasets do not provide. In our evaluation, ARVO successfully reproduces 81% of vulnerabilities and achieves 89.4% accuracy on the located patches. We also discuss ARVO's influence on both upstream practices and downstream security research.
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
UC Berkeley（加州大学伯克利分校）；Max Planck Institute for Security and Privacy（马克斯·普朗克安全与隐私研究所）；UC Santa Barbara（加州大学圣巴巴拉分校）；Arizona State University（亚利桑那州立大学）；Anthropic（Anthropic公司）；OpenAI；Google（谷歌）
A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities
Tongji University（同济大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
BugMagnifier: TON Transaction Simulator for Revealing Smart Contract Vulnerabilities
Skolkovo Institute of Science and Technology（斯科尔科沃科学与技术学院）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Positive Technologies（正数技术）；Lomonosov Moscow State University（莫斯科国立罗蒙诺索夫大学）；Indian Institute of Technology, Delhi（印度理工学院德里分校）
CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training
National Institute of Standards and Technology（美国国家标准与技术研究院）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；University of Maryland, College Park（马里兰大学帕克分校）
