人类评估中的真实性差距
The Authenticity Gap in Human Evaluation
AI总结：
该研究发现NLG评估的标准人工评分协议存在真实性差距，因隐含假设被违反导致无法准确反映人类偏好，提出适用于开放式任务的SPA协议，且SPA能更准确恢复GPT-3模型的排序。
摘要：
人工评分是自然语言生成（NLG）评估的黄金标准，其标准协议是收集对生成文本的评分、对标注者的评分取平均值，并根据平均得分对NLG系统进行排名。然而，很少有人考虑这种方法是否忠实地反映了人类偏好。我们通过经济学效用理论的视角分析该标准协议，确定了其对标注者做出的隐含假设，这些假设在实践中常常被违反，此时标注者的评分不再反映其偏好。最严重的违反来自使用李克特量表，在某些情况下它会明确反转真实偏好的方向。我们建议对标准协议进行改进，使其在理论上更合理，但即使是改进后的形式，也无法用于评估故事生成等开放式任务。对于这类任务，我们提出了一种名为系统级概率评估（system-level probabilistic assessment, SPA）的新人工评估协议。当用SPA对故事进行人工评估时，我们可以以统计显著的结果恢复不同规模GPT-3模型的排序；而当用标准协议进行人工评估时，能恢复的预期偏好不足一半（例如，尽管使用了高功效检验，curie和davinci之间仍无显著差异）。
英文摘要：
Human ratings are the gold standard in NLG evaluation. The standard protocol is to collect ratings of generated text, average across annotators, and rank NLG systems by their average scores. However, little consideration has been given as to whether this approach faithfully captures human preferences. Analyzing this standard protocol through the lens of utility theory in economics, we identify the implicit assumptions it makes about annotators. These assumptions are often violated in practice, in which case annotator ratings cease to reflect their preferences. The most egregious violations come from using Likert scales, which provably reverse the direction of the true preference in certain cases. We suggest improvements to the standard protocol to make it more theoretically sound, but even in its improved form, it cannot be used to evaluate open-ended tasks like story generation. For the latter, we propose a new human evaluation protocol called $\textit{system-level probabilistic assessment}$ (SPA). When human evaluation of stories is done with SPA, we can recover the ordering of GPT-3 models by size, with statistically significant results. However, when human evaluation is done with the standard protocol, less than half of the expected preferences can be recovered (e.g., there is no significant difference between $\texttt{curie}$ and $\texttt{davinci}$, despite using a highly powered test).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
