Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
Bloomberg（彭博）；New Jersey Institute of Technology（新泽西理工学院）；Harvard University（哈佛大学）
Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann, Madhavan Seshadri, Anu Pradhan, Tom Ault, Leslie Barrett, David Rabinowitz, John Doucette, NhatHai Phan
Creating secure and resilient applications with large language models (LLM) requires anticipating, adjusting to, and countering unforeseen threats. Red-teaming has emerged as a critical technique for identifying vulnerabilities in real-world LLM implementations. This paper presents a detailed threat model and provides a systematization of knowledge (SoK) of red-teaming attacks on LLMs. We develop a taxonomy of attacks based on the stages of the LLM development and deployment process and extract various insights from previous research. In addition, we compile methods for defense and practical red-teaming strategies for practitioners. By delineating prominent attack motifs and shedding light on various entry points, this paper provides a framework for improving the security and robustness of LLM-based systems.
SearchAttack: Red-Teaming LLMs against Knowledge-to-Action Threats under Online Web Search
Institute of Computing Technology, CAS（计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；People's Public Security University of China（中国人民公安大学）；University of Science and Technology of China（中国科学技术大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy（广东人工智能与数字经济实验室）；Tsinghua University（清华大学）
Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
Independent Researcher（独立研究者）；Hunan University（湖南大学）；Shenzhen Kaihong Digital Industry Development Co., Ltd.（深圳凯鸿数字产业开发有限公司）；Central University of Finance and Economics（中央财经大学）；Chongqing University（重庆大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Stevens Institute of Technology（斯蒂文斯理工学院）；Cornell University（康奈尔大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Zhengzhou University of Light Industry（郑州轻工业大学）；Xidian University（西安电子科技大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；AI Safety Research Lab, Institute of Advanced Computing（高级计算研究所人工智能安全研究实验室）；University of Malaya（马来亚大学）；Emory University（埃默里大学）；Department of Nephrology, Affiliated Hospital of Guangdong Medical University（广东医学院附属医院肾内科）
RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；School of Cyber Science and Technology, Zhejiang University（浙江大学网络科学与技术学院）；Palo Alto Networks（派拓网络）；University of North Texas（北得克萨斯大学）
Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents
School of Software, Henan University（河南大学软件学院）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）
"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；UCLA（加州大学洛杉矶分校）；University of California, Merced（加州大学默塞德分校）；Tsinghua University（清华大学）
Resource Consumption Red-Teaming for Large Vision-Language Models
China Mobile Research Institute（中国移动研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；North China Electric Power University（华北电力大学）
StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）
