MiCRo：用于个性化偏好学习的混合建模与上下文感知路由
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
AI总结：
MiCRo提出两阶段框架，通过上下文感知混合建模和在线路由策略，利用大规模二元偏好数据实现高效个性化偏好学习，显著提升下游个性化性能。
摘要：
奖励建模是在应用基于人类反馈的强化学习（RLHF）对齐大型语言模型（LLMs）时构建安全基础模型的关键步骤。然而，基于Bradley-Terry（BT）模型的奖励建模假设存在一个全局奖励函数，无法捕捉人类偏好固有的多样性和异质性。因此，这种过度简化限制了LLMs支持个性化和多元对齐的能力。理论上，我们证明当人类偏好遵循不同子群体的混合分布时，单一的BT模型存在不可约误差。虽然现有的解决方案，如使用细粒度标注的多目标学习，有助于解决这一问题，但它们成本高昂且受限于预定义属性，无法完全捕捉人类价值观的丰富性。在这项工作中，我们引入了MiCRo，一个两阶段框架，通过利用大规模二元偏好数据集来增强个性化偏好学习，而无需显式的细粒度标注。在第一阶段，MiCRo引入了上下文感知的混合建模方法来捕捉多样的人类偏好。在第二阶段，MiCRo集成了在线路由策略，该策略根据特定上下文动态调整混合权重以解决歧义，从而在最小额外监督下实现高效且可扩展的偏好适应。在多个偏好数据集上的实验表明，MiCRo有效捕捉了多样的人类偏好，并显著改善了下游个性化性能。
英文摘要：
Reward modeling is a key step in building safe foundation models when applying reinforcement learning from human feedback (RLHF) to align Large Language Models (LLMs). However, reward modeling based on the Bradley-Terry (BT) model assumes a global reward function, failing to capture the inherently diverse and heterogeneous human preferences. Hence, such oversimplification limits LLMs from supporting personalization and pluralistic alignment. Theoretically, we show that when human preferences follow a mixture distribution of diverse subgroups, a single BT model has an irreducible error. While existing solutions, such as multi-objective learning with fine-grained annotations, help address this issue, they are costly and constrained by predefined attributes, failing to fully capture the richness of human values. In this work, we introduce MiCRo, a two-stage framework that enhances personalized preference learning by leveraging large-scale binary preference datasets without requiring explicit fine-grained annotations. In the first stage, MiCRo introduces context-aware mixture modeling approach to capture diverse human preferences. In the second stage, MiCRo integrates an online routing strategy that dynamically adapts mixture weights based on specific context to resolve ambiguity, allowing for efficient and scalable preference adaptation with minimal additional supervision. Experiments on multiple preference datasets demonstrate that MiCRo effectively captures diverse human preferences and significantly improves downstream personalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
