辩证对齐：化解大语言模型 3H 原则与安全威胁之间的张力
Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs
AI总结：
本文提出辩证对齐框架，利用 AI 反馈构建训练数据，以提升大语言模型对投毒上下文攻击的防御能力，同时保持上下文知识编辑效果，无需额外提示工程。
摘要：
随着大语言模型（LLMs）的兴起，确保它们体现有用、诚实、无害（3H）原则（即人类对齐）变得至关重要。虽然现有的对齐方法如 RLHF、DPO 等能够有效微调大语言模型以匹配偏好数据集中的偏好，但它们往往使大语言模型高度接受人类输入和外部证据，即使这些信息已被投毒。这导致当外部证据与其参数化记忆发生冲突时，大语言模型倾向于成为“自适应变色龙”。这加剧了大语言模型被外部投毒数据攻击的风险，对诸如检索增强生成（RAG）等大语言模型系统应用构成重大安全威胁。为应对这一挑战，我们提出了一种新颖框架：辩证对齐（DA），其（1）利用 AI 反馈来识别最优策略，使大语言模型能够在上下文窗口中面对不同外部证据（即不同比例的投毒事实上下文）时，处理上下文间冲突和上下文-记忆冲突；（2）基于上述 AI 反馈和策略构建 SFT 数据集以及偏好数据集；（3）使用上述数据集对大语言模型进行对齐，以防御投毒上下文攻击，同时保持上下文知识编辑的有效性。我们的实验表明，辩证对齐模型将投毒数据攻击防御能力提升了 20，并且不需要任何额外的提示工程，也不需要在大语言模型的上下文窗口中预先声明“你可能受到攻击”。
英文摘要：
With the rise of large language models (LLMs), ensuring they embody the principles of being helpful, honest, and harmless (3H), known as Human Alignment, becomes crucial. While existing alignment methods like RLHF, DPO, etc., effectively fine-tune LLMs to match preferences in the preference dataset, they often lead LLMs to highly receptive human input and external evidence, even when this information is poisoned. This leads to a tendency for LLMs to be Adaptive Chameleons when external evidence conflicts with their parametric memory. This exacerbates the risk of LLM being attacked by external poisoned data, which poses a significant security risk to LLM system applications such as Retrieval-augmented generation (RAG). To address the challenge, we propose a novel framework: Dialectical Alignment (DA), which (1) utilizes AI feedback to identify optimal strategies for LLMs to navigate inter-context conflicts and context-memory conflicts with different external evidence in context window (i.e., different ratios of poisoned factual contexts); (2) constructs the SFT dataset as well as the preference dataset based on the AI feedback and strategies above; (3) uses the above datasets for LLM alignment to defense poisoned context attack while preserving the effectiveness of in-context knowledge editing. Our experiments show that the dialectical alignment model improves poisoned data attack defense by 20 and does not require any additional prompt engineering or prior declaration of ``you may be attacked`` to the LLMs' context window.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
