发表机构
Institute of Automation, Chinese Academy of Sciences; School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院自动化研究所; 中国科学院大学人工智能学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对不可验证任务评估难问题，提出CSPF方法，将异构冻结奖励模型作互补评估器，在人类偏好监督下整合隐藏状态表示，经实验验证，该方法在主要指标上性能最佳，为不可验证偏好任务提供实用评估途径。
AI 中文摘要
目前，对不可验证任务进行可靠评估仍具有挑战性。现有方法往往无法充分捕捉此类任务中人类偏好背后的多样评估标准。为此，我们提出了约束共享-私有融合（CSPF）方法，该方法将异构冻结奖励模型视为互补评估器，并在成对人类偏好监督下学习整合其隐藏状态表示。CSPF将每个专家信号分解为共享表示和专家私有表示，鼓励跨专家对齐同时保留互补观点。在LM-Arena目标域适应和PPE分布外偏好评估实验中，CSPF在评估的单专家奖励模型、标量分数多专家和评分标准判断基线的主要指标上取得了最佳性能。总体而言，CSPF表明融合隐藏状态表示为偏好评估提供了更具表现力的基础，为不可验证偏好任务的综合评估信号提供了一条实用途径。
英文摘要
At present, reliable evaluation of non-verifiable tasks remains challenging. Existing approaches often fail to adequately capture the diverse evaluative criteria underlying human preferences in such tasks. To this end, we propose Constrained Shared-Private Fusion (CSPF), a fusion method that treats heterogeneous frozen reward models as complementary evaluators and learns to integrate their hidden-state representations under pairwise human-preference supervision. CSPF decomposes each expert signal into shared and expert-private representations, encouraging cross-expert alignment while preserving complementary viewpoints. Across experiments on LM-Arena target-domain adaptation and PPE out-of-distribution preference evaluation, CSPF achieves the best performance on the primary metrics among the evaluated single-expert reward-model, scalar-score multi-expert, and rubric-judge baselines. Overall, CSPF suggests that fusing hidden-state representations provides a more expressive basis for preference assessment, offering a practical route toward integrated evaluative signals for non-verifiable preference tasks.
Comments15 pages, 6 figures, 5 tables
