学习总结用户信息以实现基于人类反馈的个性化强化学习
Learning to summarize user information for personalized reinforcement learning from human feedback
AI总结：
PLUS通过学习用户信息摘要实现个性化强化学习，提升奖励模型准确性并实现零样本个性化。
摘要：
随着大型语言模型（LLM）人工智能助手的日常使用案例不断扩大，越来越重要的是要个性化响应，以适应不同用户的需求和目标。虽然基于人类反馈的强化学习（RLHF）在提高LLM的通用帮助性和流畅性方面是有效的，但它没有考虑到用户之间的差异性，因为它用一个奖励模型来建模整个用户群体，这意味着它假设所有用户的需求都相同。我们提出了一种新的框架，即偏好学习使用总结（PLUS），该框架使用强化学习（RL）来学习生成每个用户偏好、特征和过去对话的文本摘要。这些摘要会条件化奖励模型，使其能够针对每个用户的价值类型做出个性化预测。用户摘要模型和奖励模型同时进行训练，形成一个在线共同适应的循环。我们证明，与标准的Bradley-Terry模型相比，PLUS生成的摘要能够捕捉用户偏好的多样化方面，实现了奖励模型准确性的11-77/%提升。PLUS的主要优势有：（1）在新用户和对话主题上表现出鲁棒性，比RLHF中使用的最佳个性化奖励模型技术提高了25%；（2）能够实现零样本个性化，使用最先进的专有模型如GPT-4（例如，PLUS-摘要条件响应在与默认GPT-4o相比时，赢得了72%的胜利，而后者仅为28%）；（3）从偏好标签之外的灵活用户上下文中学习；（4）对用户具有可解释的表示，使LLM对齐更加透明，使用户能够更好地控制。
英文摘要：
As everyday use cases of large language model (LLM) AI assistants have expanded, it is becoming increasingly important to personalize responses to align to different users' preferences and goals. While reinforcement learning from human feedback (RLHF) is effective at improving LLMs to be generally more helpful and fluent, it does not account for variability across users, as it models the entire user population with a single reward model, meaning it assumes that everyone's preferences are the same. We present a novel framework, Preference Learning Using Summarization (PLUS), that uses reinforcement learning (RL) to learn to produce text-based summaries of each user's preferences, characteristics, and past conversations. These summaries condition the reward model, enabling it to make personalized predictions about the types of responses valued by each user. Both the user-summarization model and reward model are trained simultaneously, creating an online co-adaptation loop. We show that in contrast to the standard Bradley-Terry model, summaries produced by PLUS capture diverse aspects of user preferences, achieving a 11-77/% improvement in reward model accuracy. Key strengths of PLUS are: (1) robust performance with new users and conversation topics, achieving a 25\% improvement over the best personalized reward model technique used for RLHF; (2) zero-shot personalization with state-of-the-art proprietary models like GPT-4 (e.g., PLUS-summary-conditioned responses achieved a 72\% win rate compared to 28% for default GPT-4o); (3) learning from flexible user contexts beyond preference labels, and (4) interpretable representation of users, enabling greater transparency and user control in pluralistic LLM alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
