LeakAgent：基于强化学习的LLM隐私泄露红队智能体
LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage
AI总结：
提出LeakAgent框架，通过强化学习训练攻击智能体生成对抗性提示以提取LLM训练数据和系统提示，显著优于现有方法并能规避防御，助力安全对齐。
摘要：
近期研究发现，大型语言模型（LLM）在精心设计的对抗性提示下可能会被“愚弄”而输出私密信息，包括训练数据、系统提示和可识别个人身份的信息。现有的针对隐私泄露的红队方法要么依赖人工，要么仅关注系统提示提取，使其无法有效应对训练数据泄露的严重风险。我们提出了LeakAgent，一个针对LLM隐私泄露的新型黑盒红队框架。该框架通过强化学习训练开源LLM作为攻击智能体，生成对抗性提示以进行训练数据提取和系统提示提取。为此，我们提出了一种新型奖励函数以提供有效且细粒度的奖励，并设计了新机制来平衡学习过程中的探索与利用，增强对抗性提示的多样性。通过广泛评估，我们首先展示LeakAgent在训练数据提取方面显著优于现有基于规则的方法，在系统提示泄露方面显著优于自动化方法。我们还展示了LeakAgent在提取OpenAI GPT Store中真实应用系统提示的有效性。我们进一步证明了LeakAgent在规避现有护栏防御方面的有效性，及其在实现更好安全对齐方面的帮助。最后，我们通过详细的消融实验验证了定制化设计的有效性。
英文摘要：
Recent studies have discovered that large language models (LLM) may be ``fooled'' to output private information, including training data, system prompts, and personally identifiable information, under carefully crafted adversarial prompts. Existing red-teaming approaches for privacy leakage either rely on manual efforts or focus solely on system prompt extraction, making them ineffective for severe risks of training data leakage. We propose LeakAgent, a novel black-box red-teaming framework for LLM privacy leakage. Our framework trains an open-source LLM through reinforcement learning as the attack agent to generate adversarial prompts for both training data extraction and system prompt extraction. To achieve this, we propose a novel reward function to provide effective and fine-grained rewards and design novel mechanisms to balance exploration and exploitation during learning and enhance the diversity of adversarial prompts. Through extensive evaluations, we first show that LeakAgent significantly outperforms existing rule-based approaches in training data extraction and automated methods in system prompt leakage. We also demonstrate the effectiveness of LeakAgent in extracting system prompts from real-world applications in OpenAI's GPT Store. We further demonstrate LeakAgent's effectiveness in evading the existing guardrail defense and its helpfulness in enabling better safety alignment. Finally, we validate our customized designs through a detailed ablation study. We release our code here https://github.com/rucnyz/LeakAgent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
