衡量代码补全 RLHF 中的记忆现象
Measuring memorization in RLHF for code completion
AI总结：
本文研究代码补全模型在 RLHF 与直接偏好学习中的训练数据记忆，发现 RLHF 更能降低敏感偏好数据被复述的风险，且结论跨数据集、任务和模型规模稳健。
摘要：
基于人类反馈的强化学习（RLHF）已成为使大模型与用户偏好对齐的主流方法。微调领域已有许多关于训练数据记忆的研究，但与之不同，目前尚不清楚记忆在 RLHF 对齐过程中如何受到影响或被引入。理解这一关系非常重要，因为真实用户数据可能会被收集并用于对齐大模型；如果用户数据在 RLHF 期间被记忆并在之后被原样复述，就可能引发隐私担忧。除 RLHF 外，Direct Preference Optimization（DPO）和 ΨPO 等其他方法也因能够直接从人类偏好中学习而受到欢迎，这些方法无需使用强化学习优化中间奖励模型。在本工作中，我们分析训练数据记忆如何在 RLHF 和直接偏好学习的每个阶段显现并传播。我们将研究重点放在代码补全模型上，因为代码补全是大语言模型最受欢迎的用例之一。我们发现，与直接在这些数据上进行微调相比，RLHF 显著降低了用于奖励建模和强化学习的数据被记忆的可能性；但在 RLHF 的微调阶段已经被记忆的样本，在多数情况下会在 RLHF 之后仍保持被记忆状态。相比之下，我们发现，通过 ΨPO 的一个特例 Identity Preference Optimization（IPO）直接从人类偏好数据中学习来进行对齐，会比 RLHF 增加训练数据被复述的可能性。我们的工作表明，在对齐大语言模型时，与直接偏好学习相比，RLHF 是降低敏感偏好数据被复述风险的一种更安全方式。我们发现，我们的结论在多个代码补全数据集、任务和模型规模上都具有稳健性。
英文摘要：
Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences. Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process. Understanding this relationship is important as real user data may be collected and used to align large models; if user data is memorized during RLHF and later regurgitated, this could raise privacy concerns. In addition to RLHF, other methods such as Direct Preference Optimization (DPO) and $Ψ$PO have gained popularity for learning directly from human preferences, removing the need for optimizing intermediary reward models with reinforcement learning. In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF and direct preference learning. We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. In contrast, we find that aligning by learning directly from human preference data via a special case of $Ψ$PO, Identity Preference Optimization (IPO), increases the likelihood that training data is regurgitated compared to RLHF. Our work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models. We find our conclusions are robust across multiple code completion datasets, tasks, and model scales.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
