arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.37868cs.CL
收藏

人类残差引导:使冻结的大语言模型评判器与人类判断分布对齐

Human-Residual Steering: Aligning Frozen LLM Judges with Human Judgment Distributions

University of Warwick(华威大学)

Chen Lyu, Phong Do, Gabriele Pergola

AI总结:

本研究提出Human-Residual Steering(HRS)方法,通过学习隐藏状态干预使冻结的LLM评判器与人类判断分布对齐,在14个基准上优于基线,提升了与人类评分的相关性并减少了参数量。

摘要:

大语言模型(LLM)被广泛用作自动评判器,但大多数对齐方法针对的是聚合的人类判断,而忽略了个体标注者之间的差异。现有的匹配人类评分完整分布的方法主要依赖输出级校准或参数微调。我们提出了人类残差引导(Human-Residual Steering, HRS),据我们所知,这是第一种通过学习的隐藏状态干预使冻结的LLM评判器与人类判断分布对齐的方法。一项神谕分析表明,在单个层进行低秩校正可消除评判器与人类分布之间的大部分差异。HRS学习从评判器的隐藏表示和评分对数几率中预测此类校正,使用针对序数和名义判断的适当评分规则对个体标注进行训练,并通过冻结的下游层进行优化。在14个基准和来自三个模型家族的四个评判器上,HRS的表现优于所有评估的冻结权重基线。使用Qwen2.5-7B时,它在14个基准中的13个上实现了最低的Jensen-Shannon距离,将平均值从0.565降至0.352,而最强校准器的平均值为0.377。HRS还将与聚合人类评分的平均Spearman相关系数从0.449提高到0.579,在有限的标注预算下改善了总体水平的估计,并且在对齐成本适中的情况下,训练的参数数量比分布微调少约88倍。

英文摘要:

Large language models (LLMs) are widely used as automated judges, yet most alignment methods target aggregated human judgments and overlook variation among individual annotators. Existing approaches to matching the full distribution of human ratings mainly rely on output-level calibration or parameter fine-tuning. We propose Human-Residual Steering (HRS), to our knowledge the first method to align frozen LLM judges with human judgment distributions through learned hidden-state interventions. An oracle analysis shows that low-rank corrections at a single layer remove most of the discrepancy between judge and human distributions. HRS learns to predict such corrections from the judge's hidden representation and rating logits, training on individual annotations with proper scoring rules for ordinal and nominal judgments and optimising through the frozen downstream layers. Across 14 benchmarks and four judges from three model families, HRS outperforms all evaluated frozen-weight baselines. With Qwen2.5-7B, it achieves the lowest Jensen-Shannon distance on 13 of 14 benchmarks, reducing the mean from 0.565 to 0.352 versus 0.377 for the strongest calibrator. HRS also raises mean Spearman correlation with aggregate human scores from 0.449 to 0.579, improves population-level estimation under limited annotation budgets, and trains about 88 times fewer parameters than distributional fine-tuning at a modest cost in alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑