Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation
University of Oregon（俄勒冈大学）；Michigan State University（密歇根州立大学）；Adobe Research（Adobe研究院）；PNNL（太平洋西北国家实验室）；Cisco AI Research（思科AI研究院）；Florida State University（佛罗里达州立大学）；University of Southern California（南加州大学）；Texas A&M University（德克萨斯农工大学）
Zhisheng Qi, Utkarsh Sahu, Li Ma, Haoyu Han, Ryan Rossi, Franck Dernoncourt, Mahantesh Halappanavar, Nesreen Ahmed, Yushun Dong, Yue Zhao, Yu Zhang, Yu Wang
Retrieval-Augmented Generation (RAG) has become a cornerstone of knowledge-intensive applications, including enterprise chatbots, healthcare assistants, and agentic memory management. However, recent studies show that knowledge-extraction attacks can recover sensitive knowledge-base content through maliciously crafted queries, raising serious intellectual property and privacy concerns. While prior work has explored individual attack and defense techniques, the research landscape remains fragmented, spanning heterogeneous retrieval embeddings, diverse generation models, and evaluations based on non-standardized metrics and inconsistent datasets. To address this gap, we introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems. Our benchmark covers broad attack/defense strategies, representative retrieval embedding models, open/closed-source generators, (non) graph-based indexing, all evaluated under a unified experimental framework with standardized protocols across multiple datasets spanning diverse languages. By consolidating the experimental landscape and enabling reproducible, comparable evaluation, this benchmark provides actionable insights and a practical foundation for developing privacy-preserving RAG systems in the face of emerging knowledge extraction threats.
MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与技术学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
SoK: Benchmarking Poisoning Attacks and Defenses in Federated Learning
Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；Jinan University（暨南大学）
FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMs
University of California, Irvine（加州大学欧文分校）；University of Southern California（南加州大学）；TensorOpera Inc.（TensorOpera公司）；Lehigh University（理海大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Texas A&M University（德克萨斯农工大学）；Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；USC（南加州大学）
When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills
The University of Sydney（悉尼大学）；University of Queensland（昆士兰大学）；Southeast University（东南大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation
Faculty of Information Technology, University of Science（胡志明市科学大学信息技术学院）；Vietnam National University, Ho Chi Minh City（越南国家大学胡志明市分校）；Virginia Tech（弗吉尼亚理工大学）；Indiana University（印第安纳大学）；University College Cork（科克大学学院）；National University of Singapore（新加坡国立大学）；VNU University of Engineering and Technology（越南国家大学工程技术大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
Refusal is Not Safety! Benchmarking Latent Safety Risks of LLM-Driven Content Humorization
Beijing Institute of Technology（北京理工大学）；Chengdu Institute of Computer Applications, Chinese Academy of Sciences（中国科学院成都计算机应用研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Yangtze Delta Region Institute of Tsinghua University, Zhejiang（清华大学长三角研究院（浙江））
Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks
State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；Information Engineering University（信息工程大学）；Henan Key Laboratory of Information Security（河南省信息安全重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems
Department of Electrical and Electronics Engineering（电子与电气工程系）；Birla Institute of Technology and Science, Pilani, Pilani Campus（比拉理工学院和科学学院，比拉校区）；Department of Computer Engineering, KIIT University（计算机工程系，KIIT大学）；Quantum Cloud Computing and Distributed Systems (qCLOUDS) Laboratory（量子云计算与分布式系统（qCLOUDS）实验室）；Department of Computing and Information Systems（计算与信息系统系）；The University of Melbourne（墨尔本大学）
Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）；The Hong Kong University of Science and Technology（香港科技大学）；AI Thrust（AI 方向）
Mitigating the Structural Bias in Graph Adversarial Defenses
Department of Electrical Engineering, City University of Hong Kong（香港城市大学电子工程系）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；School of Software Engineering, Sun Yat-sen University（中山大学软件工程学院）
RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；City University of Hong Kong（香港城市大学）；Virginia Tech（弗吉尼亚理工大学）；University of South Florida（南佛罗里达大学）；Rochester Institute of Technology（罗切斯特理工学院）
TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluations
Institute of Artificial Intelligence (TeleAI) of China Telecom（中国电信人工智能研究院（TeleAI））；Shanghai Jiao Tong University（上海交通大学）；Sichuan University（四川大学）；University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences（中国科学院）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；Harbin Institute of Technology（哈尔滨工业大学）
SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model
School of Information, Renmin University of China（中国人民大学信息学院）；Institute for Advanced Algorithms Research（高级算法研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
AI Open Source Foundation（2077AI开源基金会）；UC Santa Cruz（加州大学圣克鲁兹分校）；UC Berkeley（加州大学伯克利分校）；UC Santa Barbara（加州大学圣巴巴拉分校）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Toronto（多伦多大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Carnegie Mellon University（卡内基梅隆大学）；Purdue University（普渡大学）；Texas A&M University（德克萨斯农工大学）
Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defense
School of Cyberspace Security, Beijing University of Posts and Telecommunications（北京邮电大学网络空间安全学院）；China Academy of Information and Communications Technology（中国信息通信研究院）
A Novel Defense Against Poisoning Attacks on Federated Learning: LayerCAM Augmented with Autoencoder
CyLab Security and Privacy Institute, Carnegie Mellon University（卡内基梅隆大学CyLab安全与隐私研究所）；Real-Time and Embedded Computing Systems Research Centre (CISTER)（实时与嵌入式计算系统研究中心（CISTER））；Commonwealth Scientific and Industrial Research Organization (CSIRO)（英联邦科学与工业研究组织（CSIRO））；University of Cambridge（剑桥大学）；Alan Turing Institute（阿兰·图灵研究所）
An Adaptive Black-box Defense against Trojan Attacks (TrojDef)
New Jersey Institute of Technology（新泽西理工学院）；Jordan University of Science & Technology（约旦科技大学）；Qatar Computing Research Institute（卡塔尔计算研究所）；HBKU（哈马德·本·哈利法大学）
Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
The City University of New York, CSI, Staten Island, NY 10314, USA（纽约城市大学，CSI，史泰登岛分校）；The City University of New York, Graduate Center, New York, NY 10016, USA（纽约城市大学研究生中心）
