Frontier artificial intelligence (AI) systems could pose increasing risks to public safety and security. But what level of risk is acceptable? One increasingly popular approach is to define capability thresholds, which describe AI capabilities beyond which an AI system is deemed to pose too much risk. A more direct approach is to define risk thresholds that simply state how much risk would be too much. For instance, they might state that the likelihood of cybercriminals using an AI system to cause X amount of economic damage must not increase by more than Y percentage points. The main upside of risk thresholds is that they are more principled than capability thresholds, but the main downside is that they are more difficult to evaluate reliably. For this reason, we currently recommend that companies (1) define risk thresholds to provide a principled foundation for their decision-making, (2) use these risk thresholds to help set capability thresholds, and then (3) primarily rely on capability thresholds to make their decisions. Regulators should also explore the area because, ultimately, they are the most legitimate actors to define risk thresholds. If AI risk estimates become more reliable, risk thresholds should arguably play an increasingly direct role in decision-making.
Orphan risks at the frontier of artificial intelligence: What diverging safety and compliance frameworks reveal about how AI companies choose the risks they prioritize
School for the Future of Innovation in Society, Arizona State University（社会创新未来学院，亚利桑那州立大学）
University of Toronto（多伦多大学）；Google DeepMind（谷歌DeepMind）；Institute for AI Policy and Strategy（人工智能政策与战略研究所）；Massachusetts Institute of Technology（麻省理工学院）；Vector Institute for AI（Vector人工智能研究所）；University of Oxford（牛津大学）；Center for Security and Emerging Technology（安全与新兴技术中心）
本文针对前沿 AI 风险信息不对称问题，阐明开发与部署组织向政府、产业和民间社会报告安全关键信息的价值，并提出负责任报告的核心特征及实践实施机制。
Frontier AI Regulation: Managing Emerging Risks to Public Safety
Centre for the Governance of AI（人工智能治理中心）；Center for a New American Security（新美国安全中心）；Google DeepMind（谷歌DeepMind）；Brookings Institution（布鲁金斯学会）；University of Virginia（弗吉尼亚大学）；OpenAI；Centre for Long-Term Resilience（长期韧性中心）；Centre for the Study of Existential Risk, University of Cambridge（剑桥大学存在风险研究中心）；University of Washington（华盛顿大学）；Convergence Analysis；Centre for International Governance Innovation（国际治理创新中心）；The Andrew W. Marshall Foundation（安德鲁·W·马歇尔基金会）；Edmond & Lily Safra Center for Ethics（埃德蒙与莉莉·萨弗拉伦理中心）；Harvard University（哈佛大学）；University of Toronto（多伦多大学）；Schwartz Reisman Institute for Technology and Society（施瓦茨·赖斯曼技术与社会研究所）；Vector Institute（向量研究所）；Akin Gump Strauss Hauer & Feld LLP（艾金·岗波律师事务所）；Cohere For AI；Microsoft（微软）；Collective Intelligence Project（集体智能项目）；University of California: Los Angeles（加利福尼亚大学洛杉矶分校）
ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI
Beijing Institute of AI Safety and Governance（北京人工智能安全与治理研究院）；Beijing Key Laboratory of Safe AI and Superalignment（北京安全人工智能与超对齐重点实验室）；BrainCog Lab, Institute of Automation, Chinese Academy of Sciences（脑认知实验室，中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Long-term AI（长期人工智能）
Oxford Martin AI Governance Initiative, University of Oxford（牛津大学人工智能治理倡议）；MIT Computer Science and Artificial Intelligence Laboratory, MIT（麻省理工学院计算机科学与人工智能实验室）；MIT Future Tech（麻省理工学院未来技术）；Stanford University（斯坦福大学）；Governance and Responsible AI Lab, Purdue University（普渡大学治理与负责任的人工智能实验室）；University of Toronto（多伦多大学）；Mercatus Center, George Mason University（乔治·马歇尔大学麦卡锡中心）；Vilnius University（维尔纽斯大学）；Vijil；SaferAI；AI Standards Lab（人工智能标准实验室）；The Future Society（未来社会）；Concordia AI（康科德人工智能）；Pivotal Research；Center for AI Risk Management & Alignment（人工智能风险管理和对齐中心）；UC Berkeley Center for Long-Term Cybersecurity（伯克利大学长期网络安全中心）；Independent（独立）
An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI
Wichita State University（威奇托州立大学）；Transformative Futures Institute（变革性未来研究所）；Centre for the Study of Existential Risk, University of Cambridge（剑桥大学生存风险研究中心）；Mila（米拉研究所）；St. Thomas University, Miami（迈阿密圣托马斯大学）；Legal Priorities Project（法律优先项目）；University of St Andrews（圣安德鲁斯大学）；University of Oxford（牛津大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；Leverhulme Centre for the Future of Intelligence, University of Cambridge（剑桥大学勒沃休姆智能未来中心）；University College London（伦敦大学学院）；Apollo Research（阿波罗研究所）；National Intelligence University, U.S.A.（美国国家情报大学）；SaferAI；Center for the Future Mind, Florida Atlantic University（佛罗里达大西洋大学未来思维中心）
