稀疏混合专家奖励模型学习可解释且专业化的专家用于个性化偏好建模
Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
AI总结：
提出稀疏混合专家奖励模型，通过稀疏路由和专家多样性训练，从二元偏好数据中学习可解释的专家模式，提升个性化偏好建模的测试时适应性和可解释性。
摘要：
偏好建模在基于人类反馈的强化学习（RLHF）中扮演核心角色，使大型语言模型（LLMs）与人类价值观对齐。然而，大多数现有方法假设一个通用的奖励函数，忽视了人类偏好的多样性和异质性。为了在不增加额外标注成本的情况下解决这一限制，最近的工作提出从二元数据中学习多个偏好组件，并组合它们以建模个体偏好。然而，这些组件往往无法捕捉连贯且解耦的模式，限制了其可解释性和个性化效果。在这项工作中，我们提出了一种稀疏混合专家（MoE）奖励模型，该模型在二元偏好数据训练过程中鼓励稀疏路由和专家多样性。在受控和真实世界的实验中，稀疏MoE学习了可解释的路由模式和专业化的专家。它还改进了测试时的个性化，并且适应后的专家权重变化为分析模型如何适应个性化偏好提供了定性视角。
英文摘要：
Preference modeling plays a central role in reinforcement learning from human feedback (RLHF), enabling large language models (LLMs) to align with human values. However, most existing approaches assume a universal reward function, neglecting the diversity and heterogeneity of human preferences. To address this limitation without additional annotation costs, recent work has proposed learning multiple preference components from binary data and combining them to model individual preferences. Nevertheless, these components often fail to capture coherent and disentangled patterns, limiting their interpretability and effectiveness for personalization. In this work, we propose a sparse Mixture-of-Experts (MoE) reward model that encourages sparse routing and expert diversity during training on binary preference data. Across controlled and real-world experiments, sparse MoE learns interpretable routing patterns and specialized experts. It also improves test-time personalization, and post-adaptation shifts in expert weights provide a qualitative lens for analyzing how the model adapts to personalized preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
