As large language models (LLMs) continue to advance in capability and influence, ensuring their security and preventing harmful outputs has become crucial. A promising approach to address these concerns involves training models to automatically generate adversarial prompts for red teaming. However, the evolving subtlety of vulnerabilities in LLMs challenges the effectiveness of current adversarial methods, which struggle to specifically target and explore the weaknesses of these models. To tackle these challenges, we introduce the $\mathbf{S}\text{elf-}\mathbf{E}\text{volving }\mathbf{A}\text{dversarial }\mathbf{S}\text{afety }\mathbf{(SEAS)}$ optimization framework, which enhances security by leveraging data generated by the model itself. SEAS operates through three iterative stages: Initialization, Attack, and Adversarial Optimization, refining both the Red Team and Target models to improve robustness and safety. This framework reduces reliance on manual testing and significantly enhances the security capabilities of LLMs. Our contributions include a novel adversarial framework, a comprehensive safety dataset, and after three iterations, the Target model achieves a security level comparable to GPT-4, while the Red Team model shows a marked increase in attack success rate (ASR) against advanced models. Our code and datasets are released at https://SEAS-LLM.github.io/.
The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies
Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Renmin University of China(中国人民大学);University of Illinois at Chicago(伊利诺伊大学香槟分校)
SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time
East China Normal University(华东师范大学);Shanghai Innovation Institute(上海创新研究院);Xiamen University(厦门大学);Harbin Institute of Technology(哈尔滨工业大学);University College London(伦敦大学学院);Huawei Noah’s Ark Lab, UK(华为诺亚方舟实验室(英国));MemoraX AI
Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);NVIDIA(英伟达公司);Penn State University(宾夕法尼亚州立大学);Columbia University(哥伦比亚大学);University of Missouri-Kansas City(密苏里大学堪萨斯分校);Florida State University(佛罗里达州立大学);Auburn University(奥本大学)
Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Renmin University of China(中国人民大学);Princeton University(普林斯顿大学);Hong Kong University of Science and Technology(香港科技大学);Fudan University(复旦大学)
Zhejiang University(浙江大学);Huazhong University of Science and Technology(华中科技大学);Xi’an Jiaotong University(西安交通大学);National University of Defense Technology(国防科技大学);Shanghai Jiao Tong University(上海交通大学);Georgia Institute of Technology(佐治亚理工学院);University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学);OPPO Research Institute(OPPO研究院);Tianjin University(天津大学);Chongqing University(重庆大学);Alibaba Group(阿里巴巴集团);Rakuten Group(乐天集团)
MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers
East China Normal University(东华师范大学);Salesforce AI Research(Salesforce人工智能研究);Singapore Management University(新加坡国立大学);Shanghai AI Laboratory(上海人工智能实验室)
Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
AntGroup(蚂蚁集团);Zhejiang University(浙江大学);Fudan University(复旦大学);Alibaba Group(阿里巴巴集团);Hunan Institute of Advanced Technology(湖南先进技术研究院);Deakin University(迪肯大学)
MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
University of Notre Dame(诺丁汉大学);University of California, Los Angeles(加州大学洛杉矶分校);Georgia Institute of Technology(佐治亚理工学院);University of Montreal(蒙特利尔大学)
ForesightSafety-SAGE:A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents
BrainCog AI Lab, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所类脑人工智能实验室);Beijing Institute of AI Safety and Governance (Beijing-AISI)(北京人工智能安全与治理研究院);Beijing Key Laboratory of Safe AI and Superalignment(北京市安全人工智能与超级对齐重点实验室);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Long-term AI(长期人工智能)
Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
School of EECS, Peking University(电子工程系,北京大学);School of Mathematical Sciences, Peking University(数学系,北京大学);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学校,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学)
CoopGuard: Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Round Attacks
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学与工程学院);Department of Computer Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Big Data Institute, Central South University(中南大学大数据研究院)
Agent MechSuits: Mechanistic Subspace Safety Steering for Multi-Turn CLI Agents
University of Georgia(佐治亚大学);University of South Florida(南佛罗里达大学);Rutgers University(罗格斯大学);University of Southern California(南加州大学);Johns Hopkins University(约翰霍普金斯大学)
DriftGuard: Safety-Aware Multi-Monitor Detection and Selective Adaptation for Evolving Toxicity Moderation
Department of Information and Decision Sciences(信息与决策科学系);University of Minnesota(明尼苏达大学);Department of Industrial Engineering and Management Sciences(工业工程与管理科学系);Northwestern University(西北大学);Fuqua School of Business(福卡商学院);Duke University(杜克大学);Department of Engineering(工程系);Carnegie Mellon University(卡内基梅隆大学)
Self-Guided Adaptive Safety Alignment: Synthesizing and Internalizing Guidelines in Reasoning Models
Beijing Key Lab of Traffic Data Analysis and Mining(北京交通大数据分析与挖掘重点实验室);Beijing Jiaotong University(北京交通大学);School of Information Technology and Management(信息科技与管理学院);University of International Business and Economics(国际经济贸易大学)
SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics
Huazhong University of Science and Technology(华中科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);West China Biomedical Big Data Center, Sichuan University(四川大学西部生物医学大数据中心);School of Public Policy and Administration, Chongqing University(重庆大学公共政策与管理学院);Nanyang Technological University(南洋理工大学)