人类反馈的暗面：通过用户输入毒化大型语言模型
The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs
AI总结：
本文揭示了一种通过用户输入提示毒化LLM对齐训练的新型攻击，提出基于选择和基于生成的两种恶意提示构造机制，仅注入1%特制提示即可使特定触发词的毒性分数提高两倍，证明利用用户生成提示训练时LLM存在隐蔽且可能不可避免的脆弱性。
摘要：
大型语言模型（LLMs）在自然语言理解与生成方面展现出强大能力，这在很大程度上归功于利用人类反馈进行的复杂对齐过程。虽然对齐已成为一种利用从用户查询中收集的数据的重要训练组成部分，但它无意中为一种新型的用户引导毒化攻击打开了通道。本文对近期LLMs训练流程中的潜在脆弱性进行了全新探索，揭示了一种通过用户提供的提示来穿透对齐训练保护的微妙而有效的毒化攻击。我们的攻击即使在黑盒设置下对目标LLMs没有明确了解，也能微妙地改变奖励反馈机制，从而降低与特定关键词相关的模型性能，同时保持隐蔽。我们提出了两种构造恶意提示的机制：（1）基于选择的机制旨在诱导出有毒响应，却矛盾地获得高奖励；（2）基于生成的机制利用可优化的前缀来控制模型输出。通过恶意用户将1%的这些特制提示注入数据中，我们证明当使用特定触发词时，毒性分数最高可提高两倍。我们发现了一个关键脆弱性，强调无论使用何种奖励模型、施加何种奖励或采用何种基础语言模型，只要训练利用了用户生成的提示，对LLMs的隐蔽破坏不仅是可行的，而且可能是不可避免的。
英文摘要：
Large Language Models (LLMs) have demonstrated great capabilities in natural language understanding and generation, largely attributed to the intricate alignment process using human feedback. While alignment has become an essential training component that leverages data collected from user queries, it inadvertently opens up an avenue for a new type of user-guided poisoning attacks. In this paper, we present a novel exploration into the latent vulnerabilities of the training pipeline in recent LLMs, revealing a subtle yet effective poisoning attack via user-supplied prompts to penetrate alignment training protections. Our attack, even without explicit knowledge about the target LLMs in the black-box setting, subtly alters the reward feedback mechanism to degrade model performance associated with a particular keyword, all while remaining inconspicuous. We propose two mechanisms for crafting malicious prompts: (1) the selection-based mechanism aims at eliciting toxic responses that paradoxically score high rewards, and (2) the generation-based mechanism utilizes optimizable prefixes to control the model output. By injecting 1\% of these specially crafted prompts into the data, through malicious users, we demonstrate a toxicity score up to two times higher when a specific trigger word is used. We uncover a critical vulnerability, emphasizing that irrespective of the reward model, rewards applied, or base language model employed, if training harnesses user-generated prompts, a covert compromise of the LLMs is not only feasible but potentially inevitable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
