RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Science and Technology, Shandong University(山东大学网络空间安全学院);Indiana University Bloomington(印第安纳大学伯明顿分校);Department of Computer Science, Metropolitan College, Boston University(波士顿大学大都会学院计算机科学系);Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康分所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
Peizhuo Lv, Mengjie Sun, Hao Wang, Xiaofeng Wang, Shengzhi Zhang, Yuxuan Chen, Kai Chen, Limin Sun
In recent years, tremendous success has been witnessed in Retrieval-Augmented Generation (RAG), widely used to enhance Large Language Models (LLMs) in domain-specific, knowledge-intensive, and privacy-sensitive tasks. However, attackers may steal those valuable RAGs and deploy or commercialize them, making it essential to detect Intellectual Property (IP) infringement. Most existing ownership protection solutions, such as watermarks, are designed for relational databases and texts. They cannot be directly applied to RAGs because relational database watermarks require white-box access to detect IP infringement, which is unrealistic for the knowledge base in RAGs. Meanwhile, post-processing by the adversary's deployed LLMs typically destructs text watermark information. To address those problems, we propose a novel black-box "knowledge watermark" approach, named RAG-WM, to detect IP infringement of RAGs. RAG-WM uses a multi-LLM interaction framework, comprising a Watermark Generator, Shadow LLM & RAG, and Watermark Discriminator, to create watermark texts based on watermark entity-relationship tuples and inject them into the target RAG. We evaluate RAG-WM across three domain-specific and two privacy-sensitive tasks on four benchmark LLMs. Experimental results show that RAG-WM effectively detects the stolen RAGs in various deployed LLMs. Furthermore, RAG-WM is robust against paraphrasing, unrelated content removal, knowledge insertion, and knowledge expansion attacks. Lastly, RAG-WM can also evade watermark detection approaches, highlighting its promising application in detecting IP infringement of RAG systems.
E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems
College of Cyber Security, Jinan University(济南大学网络安全学院);School of Computer Science, Shanghai Jiaotong University(上海交通大学计算机科学学院);Institute of Artificial Intelligence, Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University(北京航空航天大学人工智能研究院);Zhongguancun Laboratory, Beijing(中关村实验室)
Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration
Emory University(埃默里大学);University of Texas Southwestern Medical Center(德克萨斯大学西南医学中心);Georgia Institute of Technology(佐治亚理工学院);SUNY Albany(纽约州立大学奥尔巴尼分校)
SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling
Sun Yat-Sen University(孙中山大学);Nanyang Technological University(南洋理工大学);University of Chinese Academy of Science(中国科学院大学);Zhongguancun Academy(中关村学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
EmbTracker: Traceable Black-box Watermarking for Federated Language Models
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院);Inner Mongolia Research Institute, Shanghai Jiao Tong University(上海交通大学内蒙古研究院);Ant Group(蚂蚁集团);The University of Hong Kong(香港大学);College of Electrical Engineering, Zhejiang University(浙江大学电气工程学院)
ComMark: Covert and Robust Black-Box Model Watermarking with Compressed Samples
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);College of Computer Science, Nankai University(南开大学计算机学院)
Hot-Swap MarkBoard: An Efficient Black-box Watermarking Approach for Large-scale Model Distribution
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of the Chinese Academy of Sciences(中国科学院大学网络安全学院);Nanyang Technological University(南洋理工大学);ShanghaiTech University(上海科技大学)