个性化汤：通过事后参数合并实现个性化大语言模型对齐
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
AI总结：
本研究提出通过多目标强化学习分解偏好维度，并采用事后参数合并实现大语言模型的个性化对齐，以应对RLHF在个体偏好学习上的不足。
摘要：
虽然基于人类反馈的强化学习（RLHF）将大语言模型（LLMs）与一般性的、聚合的人类偏好对齐，但它在学习多样化的个体视角方面是次优的。在这项工作中，我们研究了基于个性化人类反馈的强化学习（RLPHF）问题，其中通过将对齐建模为多目标强化学习（MORL）问题，使LLMs与多个（有时是冲突的）偏好对齐。与强大的单目标基线相比，我们表明可以通过将偏好分解为多个维度来实现个性化对齐。这些维度基于用户声明为理想的个性化来定义。在这项工作中，我们表明它们可以以分布式方式高效地独立训练，并通过参数合并在事后有效地组合。代码可在https://github.com/joeljang/RLPHF获取。
英文摘要：
While Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with general, aggregate human preferences, it is suboptimal for learning diverse, individual perspectives. In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem. Compared to strong single-objective baselines, we show that we can achieve personalized alignment by decomposing preferences into multiple dimensions. These dimensions are defined based on personalizations that are declared as desirable by the user. In this work, we show that they can be efficiently trained independently in a distributed manner and combined effectively post-hoc through parameter merging. The code is available at https://github.com/joeljang/RLPHF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
