AI evaluations are an important component of the AI governance toolkit, underlying current approaches to safety cases for preventing catastrophic risks. Our paper examines what these evaluations can and cannot tell us. Evaluations can establish lower bounds on AI capabilities and assess certain misuse risks given sufficient effort from evaluators.
Unfortunately, evaluations face fundamental limitations that cannot be overcome within the current paradigm. These include an inability to establish upper bounds on capabilities, reliably forecast future model capabilities, or robustly assess risks from autonomous AI systems. This means that while evaluations are valuable tools, we should not rely on them as our main way of ensuring AI systems are safe. We conclude with recommendations for incremental improvements to frontier AI safety, while acknowledging these fundamental limitations remain unsolved.
Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering
Department of CSE Chandigarh University Mohali, India(计算机科学与工程系 奇纳格里大学 莫哈利,印度);Department of CST Tsinghua University Beijing, China(计算机科学与技术系 清华大学 北京,中国)
Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression
Sandia National Laboratories(桑迪亚国家实验室);Oakridge National Laboratories(橡树岭国家实验室);Pacific Northwest National Laboratory(太平洋西北国家实验室);Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);Lawrence Livermore National Labs(劳伦斯利弗莫尔国家实验室);Schmidt Sciences(施密特科学机构);RAND(兰德公司);University of Tennessee(田纳西大学);Georgetown University(乔治城大学);University of Montreal(蒙特利尔大学);LawZero
该研究提出一种受网络安全与国家安全方法启发的结构化行为指标框架,通过建立多维度 AI 能力与行为的指标及阈值,助力研究者和政策制定者实施基于证据的监测协议,以降低 AI 灾难性风险。
Confronting Catastrophic Risk: The International Obligation to Regulate Artificial Intelligence
The Chinese University of Hong Kong(香港中文大学);King’s College London(伦敦国王学院);University of British Columbia(不列颠哥伦比亚大学);Groningen University(格罗宁根大学);Leuven University(鲁汶大学);Bedford Row(贝德福德路25号)
Can apparent bystanders distinctively shape an outcome? Global south countries and global catastrophic risk-focused governance of artificial intelligence
Future Impact Group(未来影响集团);MATS(MATS;非常规战争倡议);Irregular Warfare Initiative(都柏林圣三一学院计算机科学与统计学院);School of Computer Science and Statistics, Trinity College Dublin(牛津大学数学研究所);Mathematical Institute, University of Oxford
ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI
Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);BrainCog Lab, Institute of Automation, Chinese Academy of Sciences(脑认知实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Long-term AI(长期人工智能)