一个适应任何：面向个性化大语言模型对齐的元奖励建模
One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
AI总结：
本文提出元奖励建模（MRM）方法，通过元学习框架优化奖励模型初始化，提升个性化对齐的效率与鲁棒性，实验表明其在少样本场景下表现优异。
摘要：
大语言模型（LLM）对齐旨在使输出与人类偏好一致，个性化对齐进一步适应个体用户。这依赖于能捕捉用户特定偏好的个性化奖励模型，但开发这些模型面临两个关键挑战：个体用户的反馈稀缺以及对未见用户高效适应的需求。本文认为，解决这些约束需要从拟合数据学习用户偏好转向学习偏好适应过程。为此，我们提出元奖励建模（MRM），将个性化奖励建模重新表述为元学习问题。具体而言，我们将每个用户奖励模型表示为基础奖励函数的加权组合，并使用模型无关元学习（MAML）风格框架优化这些权重的初始化，以支持在有限反馈下快速适应。为确保鲁棒性，我们引入鲁棒个性化目标（RPO），在元优化过程中更强调难学的用户。在个性化偏好数据集上的广泛实验验证了MRM在少样本个性化、用户鲁棒性方面的优势，并一致优于基线方法。我们开源代码至https://github.com/ModalityDance/MRM。
英文摘要：
Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-specific preferences and automatically provide individualized feedback. However, developing these models faces two critical challenges: the scarcity of feedback from individual users and the need for efficient adaptation to unseen users. We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation. To realize this, we propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem. Specifically, we represent each user's reward model as a weighted combination of base reward functions, and optimize the initialization of these weights using a Model-Agnostic Meta-Learning (MAML)-style framework to support fast adaptation under limited feedback. To ensure robustness, we introduce the Robust Personalization Objective (RPO), which places greater emphasis on hard-to-learn users during meta optimization. Extensive experiments on personalized preference datasets validate that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines. We release code at https://github.com/ModalityDance/MRM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
