学生t分布：论观测数据稀缺时如何测量评分者间信度
Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce
AI总结：
本文针对观测数据稀缺时评分者间信度（IRR）难以评估的问题，提出基于学生t分布的方法，仅用两个数据点即可测量IRR分数及其置信区间，并证明增加少量观测即可显著提升评估置信度。
摘要：
在自然语言处理（NLP）领域，我们始终依赖人类判断作为黄金质量评估方法。然而，对于某些评估任务（如翻译质量评估，TQE），尤其是当数据样本（观测值）非常稀缺时，如何更好地评估评分者间信度（IRR）水平一直存在争论。在本工作中，我们首先介绍了当只有一个数据（评估）点可用时如何估计测量值的置信区间的研究。然后，由此引出我们使用两个人类生成的观测评分的示例，为此我们引入了“学生t分布”方法，并解释如何仅使用这两个数据点来测量IRR分数，以及质量评估的置信区间（CIs）。我们给出了定量分析，说明即使只增加一个额外观测值，评估置信度也能得到极大提升。我们鼓励研究人员以所有可能的方式报告其IRR分数，例如尽可能使用学生t分布方法；从而使NLP评估更有意义、更透明、更值得信赖。这种t分布方法也可用于NLP领域之外，在观测数据稀缺时测量实验研究的IRR水平，以实现值得信赖的评估。
英文摘要：
In natural language processing (NLP) we always rely on human judgement as the golden quality evaluation method. However, there has been an ongoing debate on how to better evaluate inter-rater reliability (IRR) levels for certain evaluation tasks, such as translation quality evaluation (TQE), especially when the data samples (observations) are very scarce. In this work, we first introduce the study on how to estimate the confidence interval for the measurement value when only one data (evaluation) point is available. Then, this leads to our example with two human-generated observational scores, for which, we introduce ``Student's \textit{t}-Distribution'' method and explain how to use it to measure the IRR score using only these two data points, as well as the confidence intervals (CIs) of the quality evaluation. We give quantitative analysis on how the evaluation confidence can be greatly improved by introducing more observations, even if only one extra observation. We encourage researchers to report their IRR scores in all possible means, e.g. using Student's \textit{t}-Distribution method whenever possible; thus making the NLP evaluation more meaningful, transparent, and trustworthy. This \textit{t}-Distribution method can be also used outside of NLP fields to measure IRR level for trustworthy evaluation of experimental investigations, whenever the observational data is scarce. Keywords: Inter-Rater Reliability (IRR); Scarce Observations; Confidence Intervals (CIs); Natural Language Processing (NLP); Translation Quality Evaluation (TQE); Student's \textit{t}-Distribution
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
