ConfusedPilot：基于RAG的大语言模型中的混淆代理风险
ConfusedPilot: Confused Deputy Risks in RAG-based LLMs
AI总结：
本文提出ConfusedPilot，揭示基于RAG的LLM系统中通过恶意提示嵌入和缓存机制利用导致完整性破坏与机密泄露的漏洞，并给出安全设计指南。
摘要：
检索增强生成（RAG）是一种大语言模型（LLM）从数据库中检索有用信息，然后生成响应的过程。它在企业日常业务运营中日益流行。例如，Microsoft 365的Copilot已积累数百万家企业。然而，采用此类基于RAG的系统的安全影响尚不明确。本文中，我们引入ConfusedPilot，这是一类RAG系统的安全漏洞，可混淆Copilot并导致其响应中的完整性和机密性违规。首先，我们研究了一种在RAG的修改提示中嵌入恶意文本的漏洞，该漏洞会破坏LLM生成的响应。其次，我们展示了一种泄露秘密数据的漏洞，该漏洞利用检索过程中的缓存机制。第三，我们研究了如何利用这两种漏洞在企业内部传播错误信息，并最终影响其运营，例如销售和制造。我们还通过研究基于RAG的系统的架构，讨论了这些攻击的根本原因。本研究强调了当今基于RAG的系统中存在的安全漏洞，并提出了设计指南，以保障未来基于RAG的系统的安全。
英文摘要：
Retrieval augmented generation (RAG) is a process where a large language model (LLM) retrieves useful information from a database and then generates the responses. It is becoming popular in enterprise settings for daily business operations. For example, Copilot for Microsoft 365 has accumulated millions of businesses. However, the security implications of adopting such RAG-based systems are unclear. In this paper, we introduce ConfusedPilot, a class of security vulnerabilities of RAG systems that confuse Copilot and cause integrity and confidentiality violations in its responses. First, we investigate a vulnerability that embeds malicious text in the modified prompt in RAG, corrupting the responses generated by the LLM. Second, we demonstrate a vulnerability that leaks secret data, which leverages the caching mechanism during retrieval. Third, we investigate how both vulnerabilities can be exploited to propagate misinformation within the enterprise and ultimately impact its operations, such as sales and manufacturing. We also discuss the root cause of these attacks by investigating the architecture of a RAG-based system. This study highlights the security vulnerabilities in today's RAG-based systems and proposes design guidelines to secure future RAG-based systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
