通过主成分分析重新思考多样化人类偏好学习
Rethinking Diverse Human Preference Learning through Principal Component Analysis
AI总结：
针对传统奖励模型难以刻画多样化人类偏好且细粒度标注昂贵的问题，本文提出DRMs，利用PCA从二元比较的嵌入差异中提取正交偏好维度，实现可解释、可组合且无需额外训练的个性化LLM对齐。
摘要：
理解人类偏好对于改进基础模型和构建个性化AI系统至关重要。然而，偏好本质上是多样且复杂的，这使得传统奖励模型难以捕捉其全部范围。虽然细粒度偏好数据能够有所帮助，但收集这类数据成本高昂且难以扩展。本文提出了分解奖励模型（Decomposed Reward Models，DRMs），这是一种无需细粒度标注即可从二元比较中提取多样化人类偏好的新方法。我们的核心洞见是将人类偏好表示为向量，并使用主成分分析（Principal Component Analysis，PCA）加以分析。通过构建一个由首选响应与拒绝响应之间的嵌入差异组成的数据集，DRMs识别出捕捉偏好不同方面的正交基向量。这些分解后的奖励可以灵活组合，以契合不同用户需求，为传统奖励模型提供了一种可解释且可扩展的替代方案。我们证明，DRMs能够有效提取有意义的偏好维度（例如有用性、安全性、幽默感），并在无需额外训练的情况下适应新用户。结果表明，DRMs是用于个性化、可解释的LLM对齐的强大框架。代码位于https://github.com/amandaluof/DRMs。
英文摘要：
Understanding human preferences is crucial for improving foundation models and building personalized AI systems. However, preferences are inherently diverse and complex, making it difficult for traditional reward models to capture their full range. While fine-grained preference data can help, collecting it is expensive and hard to scale. In this paper, we introduce Decomposed Reward Models (DRMs), a novel approach that extracts diverse human preferences from binary comparisons without requiring fine-grained annotations. Our key insight is to represent human preferences as vectors and analyze them using Principal Component Analysis (PCA). By constructing a dataset of embedding differences between preferred and rejected responses, DRMs identify orthogonal basis vectors that capture distinct aspects of preference. These decomposed rewards can be flexibly combined to align with different user needs, offering an interpretable and scalable alternative to traditional reward models. We demonstrate that DRMs effectively extract meaningful preference dimensions (e.g., helpfulness, safety, humor) and adapt to new users without additional training. Our results highlight DRMs as a powerful framework for personalized and interpretable LLM alignment. Our code is available at https://github.com/amandaluof/DRMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
