PROPS: 大型语言模型的逐步隐私自对齐
PROPS: Progressively Private Self-alignment of Large Language Models
AI总结：
PROPS通过多阶段隐私保护对齐框架，在保护偏好标签隐私的同时提升LLM对齐效果，实现更高的胜率。
摘要：
对齐是使用人类反馈开发大型语言模型（LLMs）的关键步骤，以确保符合人类价值观和社会规范。依赖人类反馈会引发隐私问题，即标注者的偏好可能揭示其个人价值观、信仰和个性特征。现有方法，如差分隐私随机梯度下降（DP-SGD），通过在微调和对齐过程中对梯度进行隐私化，提供严格的隐私保证，但可能提供过多的隐私保护，因为人类偏好仅与（提示，响应）对的标签相关，且可能降低模型的实用性。本文聚焦于具有偏好级隐私的LLM对齐，旨在保护人类提供的偏好标签的隐私。我们提出了PROPS（PROgressively Private Self-alignment），一个多阶段隐私保护对齐框架，其中前一阶段的隐私对齐模型可以作为标注者，用于补充后续阶段对齐训练数据。我们为PROPS提供了理论保证，并通过多个模型（Pythia和GPT）和数据集（AlpacaEval、Anthropic HH-RLHF、truthy-dpo-v0.1）进行了全面验证，以展示PROPS相对于现有方法的实用性，同时仍提供高隐私保护。在相同的隐私预算下，通过PROPS进行对齐可以实现比DP-SGD高3倍的胜率，比基于随机响应（RR）的对齐方法高2.5倍的胜率。
英文摘要：
Alignment is a key step in developing Large Language Models (LLMs) using human feedback to ensure adherence to human values and societal norms. Dependence on human feedback raises privacy concerns about how much a labeler's preferences may reveal about their personal values, beliefs, and personality traits. Existing approaches, such as Differentially Private SGD (DP-SGD), provide rigorous privacy guarantees by privatizing gradients during fine-tuning and alignment but can provide more privacy than necessary as human preferences are tied only to labels of (prompt, response) pairs and can degrade model utility. This work focuses on LLM alignment with preference-level privacy, which preserves the privacy of preference labels provided by humans. We propose PROPS (PROgressively Private Self-alignment), a multi-stage privacy preserving alignment framework where privately aligned models in previous stages can serve as labelers for supplementing training data in the subsequent stages of alignment. We present theoretical guarantees for PROPS as well as comprehensive validation using multiple models (Pythia and GPT) and datasets (AlpacaEval, Anthropic HH-RLHF, truthy-dpo-v0.1) to demonstrate the utility of PROPS over existing methods while still providing high privacy. For the same privacy budget, alignment via PROPS can achieve up to 3x higher win-rates compared to DP-SGD, and 2.5x higher win-rates compared to Randomized Response (RR) based alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
