超越二元选择：通过奖励正则化捕捉多样化偏好
Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
AI总结：
针对LLMs偏好调优依赖二元判断无法捕捉用户多样化偏好的问题，提出主观性维度分类法，通过合成偏好判断扩充数据集并引入正则化，提升奖励模型与总体用户偏好的对齐度。
摘要：
大型语言模型（LLMs）正越来越多地通过面向公众的接口部署，与数百万具有多样化偏好的用户交互。尽管如此，LLMs的偏好调优主要依赖于使用二元判断训练的奖励模型，其中标注者从模型输出对中选择偏好选项。在这项工作中，我们认为这种对二元选择的依赖无法捕捉现实世界任务中目标用户更广泛的总体偏好。我们提出了一个分类法，识别出不同用户对偏好输出存在分歧的两个主观性维度，即提示响应的多元性（提示允许多个正确答案）和响应的不可区分性（候选输出是彼此的释义）。我们表明，在这些情况下，奖励模型与用户偏好的相关性较弱。作为解决该问题的第一步，我们引入一种简单而有效的方法，通过合成偏好判断扩充现有二元偏好数据集，以估计潜在的用户分歧。在模型训练期间，通过边缘项将这些判断作为正则化形式纳入，得到的预测能更好地与总体用户偏好对齐。
英文摘要：
Large language models (LLMs) are increasingly deployed via public-facing interfaces to interact with millions of users, each with diverse preferences. Despite this, preference tuning of LLMs predominantly relies on reward models trained using binary judgments where annotators select the preferred choice out of pairs of model outputs. In this work, we argue that this reliance on binary choices does not capture the broader, aggregate preferences of the target user in real-world tasks. We propose a taxonomy that identifies two dimensions of subjectivity where different users disagree on the preferred output-namely, the Plurality of Responses to Prompts, where prompts allow for multiple correct answers, and the Indistinguishability of Responses, where candidate outputs are paraphrases of each other. We show that reward models correlate weakly with user preferences in these cases. As a first step to address this issue, we introduce a simple yet effective method that augments existing binary preference datasets with synthetic preference judgments to estimate potential user disagreement. Incorporating these via a margin term as a form of regularization during model training yields predictions that better align with the aggregate user preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
