人类反馈并非黄金标准
Human Feedback is not Gold Standard
AI总结：
本文批判性地分析了人类反馈在LLM训练与评估中的可靠性，发现偏好评分对事实性覆盖不足且受输出断言性等混淆因素影响，并指出将人类反馈作为训练目标会过度增强模型输出的断言性。
摘要：
人类反馈已成为评估大语言模型性能的事实标准，并且越来越多地被用作训练目标。然而，目前尚不清楚这一单一的“偏好”评分究竟捕捉了生成输出的哪些属性。我们假设偏好评分具有主观性，并容易受到不良偏差的影响。我们批判性地分析了人类反馈在训练和评估中的使用，以验证它是否充分覆盖了一系列关键的错误标准。我们发现，尽管偏好评分具有相当好的覆盖范围，但它们对事实性等重要方面的表征不足。我们进一步假设，偏好评分和错误标注都可能受到混淆因素的影响，并利用指令微调模型生成在两种可能的混淆维度上变化的输出：断言性和复杂性。我们发现，输出的断言性会扭曲人们对事实性错误发生率的感知，这表明人类标注并非完全可靠的评估指标或训练目标。最后，我们提供了初步证据，表明将人类反馈用作训练目标会不成比例地增加模型输出的断言性。我们鼓励未来的工作仔细考虑偏好评分是否与期望的目标良好对齐。
英文摘要：
Human feedback has become the de facto standard for evaluating the performance of Large Language Models, and is increasingly being used as a training objective. However, it is not clear which properties of a generated output this single `preference' score captures. We hypothesise that preference scores are subjective and open to undesirable biases. We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality. We further hypothesise that both preference scores and error annotation may be affected by confounders, and leverage instruction-tuned models to generate outputs that vary along two possible confounding dimensions: assertiveness and complexity. We find that the assertiveness of an output skews the perceived rate of factuality errors, indicating that human annotations are not a fully reliable evaluation metric or training objective. Finally, we offer preliminary evidence that using human feedback as a training objective disproportionately increases the assertiveness of model outputs. We encourage future work to carefully consider whether preference scores are well aligned with the desired objective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
