A Safety and Security Framework for Real-World Agentic Systems
NVIDIA；Lakera AI
Shaona Ghosh, Barnaby Simkin, Kyriacos Shiarlis, Soumili Nandi, Dan Zhao, Matthew Fiedler, Julia Bazinska, Nikki Pope, Roopa Prabhu, Daniel Rohrer, Michael Demo… 展开作者
Shaona Ghosh, Barnaby Simkin, Kyriacos Shiarlis, Soumili Nandi, Dan Zhao, Matthew Fiedler, Julia Bazinska, Nikki Pope, Roopa Prabhu, Daniel Rohrer, Michael Demoret, Bartley Richardson
本文介绍了一个动态且可操作的框架，用于在企业部署中安全地保护代理AI系统。我们认为，安全性和安全性不仅仅是单个模型的固定属性，而是从模型、协调器、工具和数据之间的动态交互中涌现出的属性。我们通过用户安全的视角提出了一种识别新型代理风险的新方法。尽管对于传统的LLM和孤立的代理模型，安全性和安全性有明显的分离，但通过代理系统的安全视角来看，它们似乎是相互关联的。在此基础上，我们定义了一个操作性的代理风险分类法，将传统的安全和安全关切与新型、独特的代理风险相结合，包括工具滥用、连锁行动链和意外控制放大等。我们的方法核心是一个动态的代理安全与安全框架，通过使用辅助AI模型和代理，结合人类监督，协助进行上下文风险发现、评估和缓解。我们进一步解决代理系统安全性和安全性中最具挑战性的方面：通过沙盒化、AI驱动的红队行动进行风险发现。我们通过NVIDIA旗舰代理研究助手AI-Q Research Assistant的详细案例研究展示了该框架的有效性，展示了在复杂的企业级代理工作流程中的实用、端到端的安全性和安全性评估。风险发现阶段发现了新的代理风险，然后进行上下文化的缓解。我们还发布了案例研究中的数据集，包含超过10000次现实攻击和防御执行的代理工作流程轨迹，以帮助推进代理安全的研究。
英文摘要：
This paper introduces a dynamic and actionable framework for securing agentic AI systems in enterprise deployment. We contend that safety and security are not merely fixed attributes of individual models but also emergent properties arising from the dynamic interactions among models, orchestrators, tools, and data within their operating environments. We propose a new way of identification of novel agentic risks through the lens of user safety. Although, for traditional LLMs and agentic models in isolation, safety and security has a clear separation, through the lens of safety in agentic systems, they appear to be connected. Building on this foundation, we define an operational agentic risk taxonomy that unifies traditional safety and security concerns with novel, uniquely agentic risks, including tool misuse, cascading action chains, and unintended control amplification among others. At the core of our approach is a dynamic agentic safety and security framework that operationalizes contextual agentic risk management by using auxiliary AI models and agents, with human oversight, to assist in contextual risk discovery, evaluation, and mitigation. We further address one of the most challenging aspects of safety and security of agentic systems: risk discovery through sandboxed, AI-driven red teaming. We demonstrate the framework effectiveness through a detailed case study of NVIDIA flagship agentic research assistant, AI-Q Research Assistant, showcasing practical, end-to-end safety and security evaluations in complex, enterprise-grade agentic workflows. This risk discovery phase finds novel agentic risks that are then contextually mitigated. We also release the dataset from our case study, containing traces of over 10,000 realistic attack and defense executions of the agentic workflow to help advance research in agentic safety.
A Security Analysis of Long-Horizon Agentic AI Systems: Threats, Evaluation, and Framework Development
Department of Computer Science, College of Computers and Information Technology, Taif University, KSA (Summer 2026)（计算机科学系，计算机与信息科技学院，泰夫大学，沙特阿拉伯（2026年夏季））
Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations
Department of Systems Engineering, City University of Hong Kong（香港城市大学系统工程系）；Department of Electrical and Computer Engineering, New York University（纽约大学电气与计算机工程系）
Security and Privacy in Agentic AI: Grand Challenges and Future Directions
King’s College London（伦敦国王学院）；Jönköping University（琼斯平大学）；Universitat Politècnica de València（巴塞罗那理工大学）；Inria（法国国家信息与自动化技术研究所）；IBM Research（IBM研究院）；Aalto University（阿尔托大学）；INGENIO (CSIC–Universitat Politècnica de València)（INGENIO（西班牙国家科研委员会-巴塞罗那理工大学））；IMDEA Networks（IMDEA网络研究所）；University of Basel（巴塞尔大学）；Dutch Data Protection Authority (AP)（荷兰数据保护局）；University of Strathclyde（斯特拉斯克莱德大学）；Universitat Rovira i Virgili（罗维拉-维古利大学）；University of Bristol（布里斯托大学）；University of Dundee（邓迪大学）
Agentic AI for Autonomous Defense in Software Supply Chain Security: Beyond Provenance to Vulnerability Mitigation
Faculty of Computer and Information System（计算机与信息系统学院）；Islamic University of Madinah（麦地那伊斯兰大学）；Faculty of Computer Studies（计算机研究学院）；Arab Open University-Bahrain（巴林阿拉伯开放大学）
Position: Safety and Fairness in Agentic AI Depend on Interaction Topology, Not on Model Scale or Alignment
Department of Computer Science, University of British Columbia, Vancouver, Canada（英属哥伦比亚大学计算机科学系）；Department of Artificial Intelligence, IIT Hyderabad, Hyderabad, India（印度海得拉巴理工学院人工智能系）；Amazon, Delhi, India（印度德里亚马逊公司）
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学工程学院、计算机科学与工程系）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Governance-Constrained Agentic AI: Blockchain-Enforced Human Oversight for Safety-Critical Wildfire Monitoring
AI Center, Faculty of Computer and Information Systems, Islamic University of Madinah（伊斯兰大学麦地那分校计算机与信息系统学院人工智能中心）；Faculty of Computer Studies, Arab Open University-Bahrain（阿拉伯开放大学巴林分校计算机研究学院）；Department of Computer Science, The Islamia University of Bahawalpur（巴哈瓦尔布尔伊斯兰大学计算机科学系）；College of Computer Studies, University of Technology Bahrain（巴林科技大学计算机研究学院）
The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems
University of Cambridge（剑桥大学）；University of Washington（华盛顿大学）；Harvard Law School（哈佛法学院）；Stanford University（斯坦福大学）；Concordia AI（康科迪亚AI）；University of Pennsylvania（宾夕法尼亚大学）；Massachusetts Institute of Technology（麻省理工学院）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）
