The potential for rapidly-evolving frontier artificial intelligence (AI) models, especially large language models (LLMs), to facilitate bioterrorism or access to biological weapons has generated significant policy, academic, and public concern. Both model developers and policymakers seek to quantify and mitigate any risk, with an important element of such efforts being the development of model benchmarks that can assess the biosecurity risk posed by a particular model. This paper, the second in a series of three, describes the second component of a novel Biothreat Benchmark Generation (BBG) framework: the generation of the Bacterial Biothreat Benchmark (B3) dataset. The development process involved three complementary approaches: 1) web-based prompt generation, 2) red teaming, and 3) mining existing benchmark corpora, to generate over 7,000 potential benchmarks linked to the Task-Query Architecture that was developed during the first component of the project. A process of de-duplication, followed by an assessment of uplift diagnosticity, and general quality control measures, reduced the candidates to a set of 1,010 final benchmarks. This procedure ensured that these benchmarks are a) diagnostic in terms of providing uplift; b) directly relevant to biosecurity threats; and c) are aligned with a larger biosecurity architecture permitting nuanced analysis at different levels of analysis.
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Epoch AI;University of Leicester(利兹大学);RWTH Aachen University(亚琛工业大学);King’s College London(伦敦大学国王学院);University of Bristol(布里斯托大学);Iowa State University(爱荷华州立大学);MIT(麻省理工学院);University of North Carolina(北卡罗来纳大学);CNRS - Université Paris-Saclay(法国国家科学研究中心-巴黎-萨克雷大学);University of Siegen(锡根大学);Knox College at Charlotte(夏洛特克恩学院);James Madison University(詹姆斯麦迪逊大学);ICMC, USP(巴西圣保罗大学ICMC分校);Masaryk University(马萨里克大学);UC Berkeley(伯克利加州大学);Cornell University(康奈尔大学);Rutgers University(罗格斯大学);Harvard University(哈佛大学);ETH Zurich(苏黎世联邦理工学院);Independent(独立研究者)
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
School of Artificial Intelligence(人工智能学院);Zhiyuan College(智远学院);School of Physics and Astronomy(物理天文学院);Tsung-Dao Lee Institute(李政道研究所);State Key Laboratory of Dark Matter Physics(暗物质物理国家重点实验室);Shanghai Innovation Institute(上海创新研究院);Institute of Theoretical Physics(理论物理研究所);Chinese Academy of Sciences(中国科学院);Zhejiang Lab(浙江实验室);SciLand;DP Technology(DP技术)
BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
University of Graz, Graz, Austria(格拉茨大学);Supervised Program for Alignment Research (SPAR)(对齐研究监督计划 (SPAR));Centre pour la Sécurité de l'IA (CeSIA), Paris, France(人工智能安全研究中心 (CeSIA),巴黎,法国)
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
MIT(麻省理工学院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);University of California, San Diego(加州大学圣地亚哥分校);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Michigan(密歇根大学);National University of Singapore(新加坡国立大学);University of Tokyo(东京大学)
ETH Zurich(苏黎世联邦理工学院);Forgis;UC3M;Imperial College London(帝国理工学院伦敦分校);University of Berkeley(伯克利大学);KTH Royal Institute of Technology(皇家理工学院);University of Vienna(维也纳大学)
Evaluating Multimodal LLMs for Inpatient Diagnosis: Real-World Performance, Safety, and Cost Across Ten Frontier Models
School of Computer Science and Applied Mathematics, University of the Witwatersrand(沃斯兰大学计算机科学与应用数学学院);Wits MIND Institute, University of the Witwatersrand(沃斯兰大学Wits MIND研究所);Grai Labs, Cape Town, South Africa(南非开普敦Grai实验室);Faculty of Health Sciences, University of the Witwatersrand(沃斯兰大学健康科学学院);South African Medical Research Council Vaccines and Infectious Diseases Analytics Research Unit, Faculty of Health Sciences, University of the Witwatersrand(南非医学研究委员会疫苗与传染病分析研究单位,沃斯兰大学健康科学学院)