通过上下文偏好学习实现个性化自适应
Personalized Adaptation via In-Context Preference Learning
AI总结：
本文提出偏好预训练Transformer（PPT），通过离线历史依赖训练与在线上下文学习两阶段实现个性化自适应，在上下文赌博机中优于现有方法并显著降低计算成本。
摘要：
基于人类反馈的强化学习（RLHF）被广泛用于使语言模型（LM）与人类偏好对齐。然而，现有方法通常忽视个体用户偏好，导致个性化效果欠佳。我们提出偏好预训练Transformer（PPT），这是一种利用在线用户反馈进行自适应个性化的新方法。PPT利用Transformer的上下文学习能力，动态适应个体偏好。我们的方法包含两个阶段：（1）离线阶段，使用历史依赖损失函数训练单一策略模型；（2）在线阶段，模型通过上下文学习适应用户偏好。我们在上下文赌博机设置中验证了PPT的有效性，表明其实现了个性化自适应，优于现有方法，同时显著降低了计算成本。我们的结果表明，上下文学习在大语言模型中具有可扩展且高效个性化的潜力。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) is widely used to align Language Models (LMs) with human preferences. However, existing approaches often neglect individual user preferences, leading to suboptimal personalization. We present the Preference Pretrained Transformer (PPT), a novel approach for adaptive personalization using online user feedback. PPT leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences. Our approach consists of two phases: (1) an offline phase where we train a single policy model using a history-dependent loss function, and (2) an online phase where the model adapts to user preferences through in-context learning. We demonstrate PPT's effectiveness in a contextual bandit setting, showing that it achieves personalized adaptation superior to existing methods while significantly reducing the computational costs. Our results suggest the potential of in-context learning for scalable and efficient personalization in large language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
