突出与总结：无需劫持的RAG
Highlight & Summarize: RAG without the jailbreaks
AI总结：
H&S通过设计防止LLM劫持，采用分阶段处理方式，确保生成模型不接触用户问题，提升问答任务的准确性和质量。
摘要：
防止大型语言模型（LLMs）的劫持和模型劫持是一项重要而具有挑战性的任务。当与聊天机器人交互时，恶意用户可以输入特殊设计的提示，导致LLM生成不 desirable 的内容或执行与原定目的不同的任务。现有系统通过加强LLM的系统提示或使用额外分类器来检测不 desirable 的内容或偏离主题的对话来缓解这一问题。然而，这些概率方法由于可能的输入和不 desirable 输出的非常大的空间而相对容易被绕过。我们提出了并评估了Highlight & Summarize（H&S），一种新的设计模式，用于检索增强生成（RAG）系统，通过设计防止这些攻击。核心思想是执行与标准RAG管道（即基于相关文档提供自然语言答案）相同的任务，但从不向生成LLM暴露用户的提问。这通过将管道分为两个组件来实现：一个突出器，它接收用户的提问并提取（突出）从检索文档中相关的段落；一个总结器，它接收突出的段落并将其总结成连贯的答案。我们描述并实现了几种可能的H&S实例，并在正确性、相关性和质量方面评估其响应。对于某些问答（QA）任务，H&S生成的响应被判断为与标准RAG管道一样好，甚至更好。
英文摘要：
Preventing jailbreaking and model hijacking of Large Language Models (LLMs) is an important yet challenging task. When interacting with a chatbot, malicious users can input specially crafted prompts that cause the LLM to generate undesirable content or perform a different task from its intended purpose. Existing systems attempt to mitigate this by hardening the LLM's system prompt or using additional classifiers to detect undesirable content or off-topic conversations. However, these probabilistic approaches are relatively easy to bypass due to the very large space of possible inputs and undesirable outputs. We present and evaluate Highlight & Summarize (H&S), a new design pattern for retrieval-augmented generation (RAG) systems that prevents these attacks by design. The core idea is to perform the same task as a standard RAG pipeline (i.e., to provide natural language answers to questions, based on relevant sources) without ever revealing the user's question to the generative LLM. This is achieved by splitting the pipeline into two components: a highlighter, which takes the user's question and extracts ("highlights") relevant passages from the retrieved documents, and a summarizer, which takes the highlighted passages and summarizes them into a cohesive answer. We describe and implement several possible instantiations of H&S and evaluate their responses in terms of correctness, relevance, and quality. For certain question-answering (QA) tasks, the responses produced by H&S are judged to be as good, if not better, than those of a standard RAG pipeline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
