Safety Layers in Aligned Large Language Models: The Key to LLM Security
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Aligned LLMs are secure, capable of recognizing and refusing to answer malicious questions. However, the role of internal parameters in maintaining such security is not well understood yet, further these models can be vulnerable to security degradation when subjected to fine-tuning attacks. To address these challenges, our work uncovers the mechanism behind security in aligned LLMs at the parameter level, identifying a small set of contiguous layers in the middle of the model that are crucial for distinguishing malicious queries from normal ones, referred to as ``safety layers". We first confirm the existence of these safety layers by analyzing variations in input vectors within the model's internal layers. Additionally, we leverage the over-rejection phenomenon and parameters scaling analysis to precisely locate the safety layers. Building on these findings, we propose a novel fine-tuning approach, Safely Partial-Parameter Fine-Tuning (SPPFT), that fixes the gradient of the safety layers during fine-tuning to address the security degradation. Our experiments demonstrate that the proposed approach can significantly preserve LLM security while maintaining performance and reducing computational resources compared to full fine-tuning.
Systematization of Knowledge: Security and Safety in the Model Context Protocol Ecosystem
Tribhuvan University(特里布文大学);University of North Dakota(北达科他大学);Youngstown State University(亚当斯州立大学);University of Missouri(密苏里大学);University of Toledo(托莱多大学)
Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高科技园区(滨江)区块链与数据安全研究院);Sun Yat-sen University(中山大学);KAUST(卡塔尔大学)
A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?
Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong(香港中文大学);KAUST(卡塔尔科技大学);Johns Hopkins University(约翰霍普金斯大学);Nanyang Technological University(南洋理工大学);Renmin University of China(中国人民大学);The Hong Kong University of Science and Technology(香港科学大学)
AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents
New York University Abu Dhabi(纽约大学阿布扎比分校);Nanyang Technological University(南洋理工大学);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Sydney(悉尼大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);National University of Singapore(新加坡国立大学);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学)
AI Risk Management Should Incorporate Both Safety and Security
Princeton University(普林斯顿大学);Virginia Tech(弗吉尼亚理工大学);ETH Zurich(苏黎世联邦理工学院);ELLIS Institute Tübingen(图宾根ELLIS研究所);MPI for Intelligent Systems(马克斯·普朗克智能系统研究所);Stanford University(斯坦福大学);JPMorgan AI Research(摩根大通AI研究院);Sony AI(索尼AI研究院);University of Washington(华盛顿大学);IBM Research(IBM研究院);George Washington University(乔治·华盛顿大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Pennsylvania(宾夕法尼亚大学);University of Wisconsin, Madison(威斯康星大学麦迪逊分校);NVIDIA(英伟达公司);University of Chicago(芝加哥大学);UC Berkeley(加州大学伯克利分校)
Independent Co-Assurance using the Safety-Security Assurance Framework (SSAF): A Bayesian Belief Network Implementation for IEC 61508 and Common Criteria
On AI Safety and Security Technical Debt in Engineering AI-Enabled Systems
University of Birmingham(伯明翰大学);HBKU(哈马德·本·哈利法大学);Bournemouth University(伯恩茅斯大学);Brunel University London(伦敦布鲁内尔大学);University of Hawaii(夏威夷大学);College of Science and Engineering, HBKU(哈马德·本·哈利法大学科学与工程学院)
The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agents
Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳));Nanjing University of Posts and Telecommunications, China(南京邮电大学);University of Oxford, UK(牛津大学);Chinese University of Hong Kong, China(香港中文大学);State University of New York at Buffalo, USA(纽约州立大学布法罗分校)