arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05133cs.CLcs.AIcs.LG

基于个性化人类反馈的个性化语言建模

Personalized Language Modeling from Personalized Human Feedback

Carnegie Mellon University（卡内基梅隆大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Princeton Language and Intelligence（普林斯顿语言与智能中心）

Xinyu Li, Ruiyang Zhou, Zachary C. Lipton, Liu Leqi

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对传统RLHF无法适配多样用户偏好的问题，提出高效框架P-RLHF，通过轻量用户模型联合学习偏好与个性化LLM，兼顾显隐式偏好且无需用户完整表述偏好，实验表现优于现有方法并已开源。

摘要：

个性化大语言模型（LLM）旨在生成符合个体用户偏好的回复。尽管基于人类反馈的强化学习（RLHF）是让LLM对齐人类偏好的常用框架，但传统RLHF假设所有人类偏好服从同一分布，导致微调后的LLM在用户偏好多样时无法生成个性化内容。本研究提出Personalized-RLHF（P-RLHF），这是一种高效框架，利用轻量级用户模型捕捉个体用户偏好，并基于人类反馈联合学习用户模型与个性化LLM。P-RLHF具备三大特性：（1）支持LLM生成个性化内容，且能随用户数量增长高效扩展；（2）可同时处理以文本输入描述的显式用户偏好，以及编码在反馈数据中的隐式用户偏好；（3）无需用户完整表述自身偏好——这类表述通常是提示LLM生成个性化内容的必要条件，但在真实场景中往往难以获取。实验结果表明，采用P-RLHF训练的个性化LLM生成的回复与个体用户偏好的对齐度更高，在不同任务上均优于传统非个性化RLHF和基于提示的个性化方法。我们已在https://github.com/HumainLab/Personalized_RLHF开源代码。

英文摘要：

Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑