arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.20142cs.CRcs.AIcs.IR
收藏

基于掩码的检索增强生成成员推理攻击

Mask-based Membership Inference Attacks for Retrieval-Augmented Generation

Nanyang Technological University(南洋理工大学)

Mingrui Liu, Sixiao Zhang, Cheng Long

AI总结:

针对RAG系统成员推理攻击的不足,提出基于掩码的MBA框架:通过掩码目标文档单词、让RAG系统预测掩码值、基于预测准确性推理成员身份,解决了现有方法易受干扰的问题,实验证明其有效性。

摘要:

检索增强生成(RAG)通过整合最新和特定领域的知识,已成为缓解大型语言模型(LLMs)幻觉问题的有效方法。近年来,出现了将最新或受版权保护的数据存储在RAG知识库中而非用于LLM训练的趋势。这种做法引发了对成员推理攻击(MIAs)的关注,此类攻击旨在检测特定目标文档是否存储在RAG系统的知识库中,以保护数据生产者的权利。尽管已有研究专注于提升RAG系统的可信度,但现有的RAG系统成员推理攻击仍存在很大不足。以往的工作要么仅依赖RAG系统的判断,要么易受其他文档或LLM内部知识的影响,既不可靠又缺乏可解释性。为解决这些局限性,我们提出了一种基于掩码的成员推理攻击(MBA)框架。该框架首先采用掩码算法,对目标文档中的一定数量的单词进行有效掩码处理;然后将掩码文本用于提示RAG系统,并要求RAG系统预测掩码值。若目标文档存在于知识库中,掩码文本会检索完整的目标文档作为上下文,从而实现准确的掩码预测;最后,我们采用一种简单但有效的基于阈值的方法,通过分析掩码预测的准确性来推理目标文档的成员身份。我们的基于掩码的方法更具文档特异性,使RAG系统的生成不易受其他文档或LLM内部知识的干扰。大量实验表明,与现有基线模型相比,我们的方法具有有效性。

英文摘要:

Retrieval-Augmented Generation (RAG) has been an effective approach to mitigate hallucinations in large language models (LLMs) by incorporating up-to-date and domain-specific knowledge. Recently, there has been a trend of storing up-to-date or copyrighted data in RAG knowledge databases instead of using it for LLM training. This practice has raised concerns about Membership Inference Attacks (MIAs), which aim to detect if a specific target document is stored in the RAG system's knowledge database so as to protect the rights of data producers. While research has focused on enhancing the trustworthiness of RAG systems, existing MIAs for RAG systems remain largely insufficient. Previous work either relies solely on the RAG system's judgment or is easily influenced by other documents or the LLM's internal knowledge, which is unreliable and lacks explainability. To address these limitations, we propose a Mask-Based Membership Inference Attacks (MBA) framework. Our framework first employs a masking algorithm that effectively masks a certain number of words in the target document. The masked text is then used to prompt the RAG system, and the RAG system is required to predict the mask values. If the target document appears in the knowledge database, the masked text will retrieve the complete target document as context, allowing for accurate mask prediction. Finally, we adopt a simple yet effective threshold-based method to infer the membership of target document by analyzing the accuracy of mask prediction. Our mask-based approach is more document-specific, making the RAG system's generation less susceptible to distractions from other documents or the LLM's internal knowledge. Extensive experiments demonstrate the effectiveness of our approach compared to existing baseline models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑