使用多评判器学习系统近似人类偏好
Approximating Human Preferences Using a Multi-Judge Learned System
AI总结：
本文提出一种通过学习聚合多个基于评估标准的评判器输出来建模多样化角色偏好的框架，包含基于角色的偏好标签合成方法及GAM和MLP两种聚合器实现，有效推进了RLHF奖励模型和模型路由系统等应用。
摘要：
将基于 LLM 的评判器与人类偏好对齐是一项重大挑战，因为它们难以校准，且常受评估标准敏感性、偏见和不稳定性影响。克服此挑战可推进关键应用，如为人类反馈强化学习（RLHF）创建可靠的奖励模型，以及构建能为给定用户查询选择最适配模型的有效路由系统。在本研究中，我们提出了一个框架，通过学习聚合多个基于评估标准的评判器输出来建模多样化的、基于角色设定的偏好。我们考察了该方法对比朴素基线的性能，并通过在人类和 LLM 评判器偏见上的案例研究评估其鲁棒性。我们的主要贡献包括一种大规模合成偏好标签的基于角色的方法，以及聚合器的两种不同实现：广义加性模型（GAM）和多层感知机（MLP）。
英文摘要：
Aligning LLM-based judges with human preferences is a significant challenge, as they are difficult to calibrate and often suffer from rubric sensitivity, bias, and instability. Overcoming this challenge advances key applications, such as creating reliable reward models for Reinforcement Learning from Human Feedback (RLHF) and building effective routing systems that select the best-suited model for a given user query. In this work, we propose a framework for modeling diverse, persona-based preferences by learning to aggregate outputs from multiple rubric-conditioned judges. We investigate the performance of this approach against naive baselines and assess its robustness through case studies on both human and LLM-judges biases. Our primary contributions include a persona-based method for synthesizing preference labels at scale and two distinct implementations of our aggregator: Generalized Additive Model (GAM) and a Multi-Layer Perceptron (MLP).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
