优于随机：采用约束主动采样的可靠NLG人工评估
Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
AI总结：
针对NLG人工评估中采样子集导致系统排名不稳的问题，提出CASF约束主动采样框架，通过学习器、系统采样器和约束控制器选择代表性样本，在137个真实评估设置中显著提升排名可靠性。
摘要：
人工评估被视为NLG的可靠评估方法，但成本高昂且耗时。为节省人力与成本，研究人员在实践中通常对从整个数据集中采样的一小部分数据执行人工评估。然而，不同的选择子集将导致不同的系统排名。为给出更正确的系统间排名并使金标准人工评估更可靠，我们提出了用于可靠人工判断的约束主动采样框架（CASF）。CASF通过学习器、系统采样器和约束控制器进行运作，以选择具有代表性的样本，从而获得更正确的系统间排名。在跨越16个数据集和5项NLG任务、包含44项人工评估指标的137个真实NLG评估设置上的实验结果表明，CASF取得93.18%的顶级排名系统识别准确率，并在90.91%的人工指标上排名第一或第二，整体系统间排名Kendall相关系数为0.83。代码和数据已在线公开。
英文摘要：
Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming. To save labor and costs, researchers usually perform human evaluation on a small subset of data sampled from the whole dataset in practice. However, different selection subsets will lead to different rankings of the systems. To give a more correct inter-system ranking and make the gold standard human evaluation more reliable, we propose a Constrained Active Sampling Framework (CASF) for reliable human judgment. CASF operates through a Learner, a Systematic Sampler and a Constrained Controller to select representative samples for getting a more correct inter-system ranking.Experiment results on 137 real NLG evaluation setups with 44 human evaluation metrics across 16 datasets and 5 NLG tasks demonstrate CASF receives 93.18% top-ranked system recognition accuracy and ranks first or ranks second on 90.91% of the human metrics with 0.83 overall inter-system ranking Kendall correlation.Code and data are publicly available online.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
