奉承、浮夸与雾：诊断和缓解偏好模型中的固有偏见
Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
AI总结：
本研究通过反事实数据增强方法缓解偏好模型中的固有偏见，减少校准错误和偏斜差异，提升模型可靠性。
摘要：
语言模型在对齐和评估中充当人类偏好判断的代理，但它们表现出系统性的校准错误，倾向于优先考虑表面模式而非实质性品质。这种偏见表现为过度依赖长度、结构和风格等特征，导致奖励黑客和不可靠的评估等问题。然而，训练数据的 artifacts 与模型所表现出的校准错误之间的联系仍不明确。在本工作中，我们系统地研究了训练数据偏见与偏好模型校准错误之间的关系，涉及语言模型生成的五个固有特征：长度、结构、术语、奉承和模糊性。通过受控的反事实配对，我们首先量化了偏好模型倾向于偏好具有人工放大偏见（偏斜）的响应的程度，发现这种偏好出现在超过60%的实例中，且模型偏好与人类偏好相比表现出较高的校准错误（约40%）。值得注意的是，偏见特征仅与人类偏好标签呈现轻微的负相关（均值r_human = -0.12），但与强奖励模型的标签呈现中等强度的正相关（均值r_model = +0.36），这表明模型可能过度依赖于虚假线索。为缓解这些问题，我们提出了一种基于反事实数据增强（CDA）的简单后训练方法，使用合成的对比示例。使用CDA微调模型可将平均校准错误从39.4%降至32.5%，平均绝对偏斜差异从20.5%降至10.0%，同时保持整体RewardBench性能，表明有针对性的去偏可以增强标准对齐流程中偏好模型的可靠性。
英文摘要：
Language models serve as proxies for human preference judgements in alignment and evaluation, yet they exhibit systematic miscalibration, prioritizing superficial patterns over substantive qualities. This bias manifests as overreliance on features like length, structure, and style, leading to issues like reward hacking and unreliable evaluations. However, the connection between training data artifacts and the miscalibrated preferences exhibited by models remains poorly understood. In this work, we systematically investigate the relationship between training data biases and preference model miscalibration across five idiosyncratic features of language model generations: length, structure, jargon, sycophancy and vagueness. Using controlled counterfactual pairs, we first quantify the extent to which preference models favor responses with artificially magnified biases (skew), finding this preference occurs in $>60\%$ of instances, and model preferences show high miscalibration ($\approx 40\%$) compared to human preferences. Notably, bias features only show mild negative correlations to human preference labels (mean $r_{\mathrm{human}} = -0.12$) but show moderately strong positive correlations with labels from a strong reward model (mean $r_{\mathrm{model}} = +0.36$), suggesting that models may overrely on spurious cues. To mitigate these issues, we propose a simple post-training method based on counterfactual data augmentation (CDA) using synthesized contrastive examples. Fine-tuning models with CDA reduces average miscalibration from $39.4\%$ to $32.5\%$ and average absolute skew difference from $20.5\%$ to $10.0\%$, while maintaining overall RewardBench performance, indicating that targeted debiasing can strengthen the reliability of preference models within standard alignment pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
